Introduction/Purpose

This report provides recommendations of the State Workforce Investment Board (SWIB) System Integration (SI) Subcommittee for implementation of system indicators across New York State Local Workforce Investment Areas. The plan builds upon the report entitled “System Indicators, Draft Report (For Review & Comment), 11/12/2002”which was adopted by the SWIB.

The plan also reflects recommendations from two meetings of State Partners, convened at the request of the SI Subcommittee to address how the required data elements in the “System Indicator Report” will be provided at the local level. The meetings were held on June 24, 2003 and August 12, 2003 and Larry Good from the Center for a Skilled Workforce facilitated both meeting.

Implementation of system indicators is intended to measure how well the workforce system is developing, integrating, continuously improving and responding to community needs, beyond program measures already established by federal agencies. For this initial stage of implementation, the plan focuses on compiling and understanding the underlying data necessary to develop the following system indicators:

1. Market Penetration Indicators
   
   1. Business Customer Market Penetration Indicator
      
   2. Individual Customer Market Penetration Indicator
2. Business Customer Repeat Usage Indicator
   
3. Total System Investment Indicator
   
4. Customer Satisfaction Indicator

Operating Principles

In defining the system indicators and developing the implementation plan, the following operating principles were established and adhered to:

1. The recommended system indicators are a starting point for working towards and developing more refined indicators of system performance.
   
2. Focus should be Statewide local measures, and should reinforce “system”, not “center” or “program.”
   
3. The recommended indicators must be relevant to the State and Local Workforce Investment Boards (LWIBs), Chief Local Elected Officials and system partners.
   
4. The two primary customers of the system are:
   
   • Businesses; and
     
   • Individuals (e.g., emerging, transitional and incumbent workers).
5. There must be a strong commitment by all local areas to supply the core data to a common database (electronically or otherwise), in order to establish consistency in measuring and interpreting indicators.
   
6. The data collection process to support calculation of the indicators must not be onerous on system partners.
   
7. The indicators should lend themselves to a system report card.
   
8. Report cards should provide a local context (e.g., local economic conditions) for the indicators.
   
9. The indicators should promote system integration and continuous improvement.
   
10. The indicators should be designed to highlight success rather than failure.
    
11. All system partners would share system outcomes (successes).

Implementation Plans

1. Market Penetration Indicators

   The ability to measure the market penetration rate for both business customers and individual customers is a fundamental goal of the workforce investment system. Furthermore, development of market penetration indicators would require partners to begin sharing common customer information, which in turn would promote integration.

   Implementation issues include:
   

   1. The market penetration outcome (rate) is likely to be considered “very small.”
      
      1. Giving meaning to the outcome, given that it is likely to be considered “very small.”
         
      2. Understanding what influences the outcome, as well as identifying “acceptable levels” rate for local workforce areas.
   2. Obtaining unduplicated data (vs. duplicated data)
      
      1. At the onset data collection should focus on getting unduplicated records, even if that means smaller numerators (i.e., outcomes).
   1. Business Market Penetration
      1. Indicator = Business Customers
                           Total Number of Businesses

      2. Definitions and Data Sources
         1. Business Customers – Total number of businesses in the local area (best unduplicated count) that have received a One-Stop service/activity in the last three years. Businesses listed in OSOS (or other common database tools).
            
         2. Total Number of Businesses – Total number of businesses in the local area that have at least one employee. This includes businesses that do not pay into the UI system (e.g. Federal government and railroads). NYSDOL data (ES202); will be generated centrally quarterly and disseminated to LWIBs.
      3. Implementation Issues
         1. Definition of One-Stop Business Services – Should consider using the definition of One-Stop Business Services outlined in Workforce Development System Technical Advisory 01-13, Core Services for Business as recommended by the WIA State Agency Partner Table, June 13, 2001 (www.workforcenewyork.org/ta/ta01-12.html).

            The intent is to capture one-stop business relationships, not merely cold call contacts (e.g., mass mailings).

            Create a workgroup to examine what services are collected now in OSOS to assess adequacy or possible needs to modify.
            

         2. Potential business exclusions – Business who directly enter their job listings into America's Job Bank; their listings will show up in OSOS via AJB but they won't have an employer record as they do if approached by staff. Issue of losing counts or getting duplicates, depending on how handled.

      4. Implementation Steps
         1. May through June 2004 – Convene workgroup to define One-Stop Business Services.
            
         2. July through September 2004 – Develop protocols to collect and compile sufficient data to compute a preliminary Business Customer Market Penetration Rate for the period ending September 30, 2004.
            
         3. October 2004 – Request and compile data from local areas, and compute a preliminary Business Customer Market Penetration Rate for the period ending September 30, 2004.
            
         4. November 2004 – Convene SI Subcommittee to review the results and determine the next steps.
            
         5. January 2004 – Request and compile data from local areas, and compute a preliminary Business Customer Market Penetration Rate for the period ending December 31, 2004.
            
         6. February / March 2004 – Convene SI Subcommittee to review the results and determine the next steps.

   2. Individual Customer Market Penetration
      1. Indicator = Individual Customers
                           Local Civilian Labor Force

      2. Definitions and Data Sources (Phase 1)
         1. Individual customers – Those who have received an employment-related service by a mandated partner. Those with swipe cards plus those receiving program services at locations other than an official one stop center.
            
         2. Local Civilian Labor Force – Those in local labor force reported by NYSDOL (BLS/LMI data).
      3. Implementation Issues
         1. The initial implementation phase will be limited to individuals for whom data can be reasonably obtained. As such, the initial phase will rely on data from local area One Stop swipe cards systems. Supplemental data may be used only if local partners can sort how to match to ensure no duplications.
            
         2. The longer-term intent is to count all customers using services. This will require data tape matching and common data tools to ensure large-scale unduplicated counts.
      4. Implementation Steps
         1. May through September 2004 – Develop protocols to collect and compile sufficient data to compute a preliminary Individual Customer Market Penetration Rate for the period ending September 30, 2004.
            
         2. October 2004 – Request and compile data from local areas, and compute a preliminary Individual Customer Market Penetration Rate for the period ending September 30, 2004.
            
         3. November 2004 – Convene SI Subcommittee to review the results and determine the next steps.
            
         4. January 2004 – Request and compile data from local areas, and compute a preliminary Individual Customer Market Penetration Rate for the period ending December 31, 2004.
            
         5. February / March 2004 – Convene SI Subcommittee to review the results and determine the next steps.

2. Business Customer Repeat Usage

   Customer Repeat Usage came out of discussion regarding Customer Satisfaction as a system indicator. Repeat usage for the business customers is viewed as reasonable starting-point for a business customer satisfaction indicator. Measuring business customer repeat usage as an indicator would also require partners to begin sharing business customer information, which in turn would promote system integration.

   1. Indicator = Repeat Business Customers
                        Business Customers

   2. Definitions and Data Sources
      1. Repeat Business Customers – Business customers who use the One Stop system more than one time during a three-year period.
         
      2. Business Customers – Same as under the market penetration indicator.
   3. Implementation Issues
      1. Intent is to count number of “events” of service, not counting subparts of a common package of service as multiple uses.
         
      2. Need to segregate job orders from other services and look at effects of America's Job Bank self-service listings on the OSOS data.
   4. Implementation Steps
      1. July through September 2004 – Develop protocols to collect and compile repeat business customer data for the period ending September 30, 2004.
         
      2. October 2004 – Request and compile data from local areas, and compute a preliminary Repeat Business Customer Usage indicator for the period ending September 30, 2004.
         
      3. November 2004 – Convene SI Subcommittee to review the results and determine the next steps.
         
      4. January 2004 – Request and compile data from local areas, and compute a preliminary Repeat Business Customer Usage indicator for the period ending December 31, 2004.
         
      5. February / March 2004 – Convene SI Subcommittee to review the results and determine the next steps.

3. Total System Investment

   The ability to measure and assess total “system investment” is a fundamental workforce system indicator. It would also require partners to begin sharing “investment” information, which in turn would promote system integration.

   Focus on Total System Investment as a system indicator came out of discussion regarding Return on Investment (ROI) as a system indicator. However, it was recognized that the methodology and underlying assumptions for computing ROI are highly subjective, which precludes use of ROI as an objective system indicator.

   1. Implementation Steps
      1. May through October 2004 – Assess the resource-mapping project prepared by NYATEP and Center for Urban Future to map resources at statewide level.
         
      2. November 2004 – Convene SI Subcommittee to review the assessment and determine the next steps.

4. Customer Satisfaction
   1. Implementation Steps
      1. May through October 2004 – Conduct an assessment of the appropriateness of using the American Customer Satisfaction Index (ACSI) as an indicator, including scope, methodology, data collection and analysis and budget.
         
      2. November 2004 – Convene SI Subcommittee to review the results and determine the next steps.