Debra Giordano - Tompkins County, Captain

Alice Savino - Herkimer/Madison/Oneida

Butch Rehm - Saratoga/Warren/Washington

Danielle DeMatteis - Yonkers

David See - Niagara

Ed Kenny - Hempstead

Gail Breen - Fulton/Montgomery/Schoharie

Gene Faber - Oyster Bay

James Calnon - Clinton/Essex/Franklin/Hamilton

Joan Sinclair - Cattaraugus/Allegany

Judy Davison - Cayuga/Cortland

Karen Springmeier - Finger Lakes

Kevin Price - Chenango/Delaware/Otsego

Laura Quigley - Sullivan

Martin Delsignore - Jefferson/Lewis

Mary Alane Wiltse - Columbia/Greene

Pam Weisberg - Rockland

Introduction

The System Indicators Team (SIT) and SIT subcommittee met to: 1) establish operating principles for defining systems indicators; and 2) to define system indicators based on the principles. This report presents background information on Meetings and Input Gathering, as well as the meeting discussion regarding Defining Systems Indicators and Establishing Operating Principles.

Meetings & Input Gathering

The following are SIT and Local Workforce Investment Board (LWIB) meetings and information gathering efforts:

July 16 & 17, 2002 – SIT meeting, Alexandria Bay, NY. SIT subcommittee formed.
July 29, 2002 – Notes from the July 16 & 17 SIT meeting posted on Web-Board (Attachment 1 presents meeting notes and the timetable).
July 30, 2002 to date – Gathering feedback from local area Partners' Table regarding system reporting (Attachment 2 presents local area comments).
August 13, 2002 – SIT subcommittee conference call.
August 16, 2002 – SIT subcommittee meeting in Utica, New York (Attachment 3 presents the meeting agenda).
September 6, 2002 to date – Notes from SIT subcommittee meeting work posted on Web-Board (Attachment 4 presents the Subcommittee meeting notes)
September 6, 2002 to date – Comments on SIT received via the Web-Board (Attachment 5 presents comments received).
October 4, 2002 – SIT subcommittee conference call.
October 22, 2002 – SIT subcommittee meeting in Buffalo, NY
November 06, 2002 – Preliminary Draft Report sent to SIT for review and comment
November 08, 2002 – SIT Conference call to discuss the Preliminary Draft Report.
November 12, 2002 – SIT Draft Report completed

Defining System Indicators

The following measures were identified through SIT and LWIB meetings and information gathering as potential system indicators:

1. Market Penetration

2. Market Share

3. Market Diversity

4. Jobseeker Entered Employment

5. Employment Retention (30 days, 90 days, 6 months)

6. Wage Levels, Gains & Replacement

7. Job Order Fill rate

8. Number of job openings in the local area and the area’s ability to fill the jobs

9. Job order fill rate versus job opening fill rate

10. Job placements

11. Reduced Jobseeker Recidivism

12. Job placements, identified by type and fill rate for job orders (should also be broken down in this area by general #'s as well as by employment type (sectors)

13. Return on Investment (ROI)

14. Total System Investment/Costs

15. Cost per customer served

16. Cost per entered employment

17. Business Customer Satisfaction

18. Jobseeker Customer Satisfaction

19. Level of importance to customers

20. Number and types of complaints, compliments and suggestions from customers and staff, as well as responses

21. Staff capacity building sessions conducted in response to complaints, compliments and suggestions.

22. Number of people trained

23. Kinds of jobs filled/trained-for

24. Cycle time

25. GEDs obtained

26. Customer Repeat Usage

27. Customer Retention Rates

28. Business Usage & Growth

29. Jobseeker Usage & Growth

30. Usage Growth Rate (last year versus this year)

31. Repeat Business Usage

32. Business Referrals

33. Customer Referrals

34. Public/Community benefit

35. Reduction in public subsidy usage (e.g., SSI, Welfare roles

36. Shorter Unemployment Insurance duration

37. Reduced unemployment rates

38. Expansion of Economic base

39. System responsiveness

40. Service continuum – other income supports

41. Use of the workforce system

42. Workforce Retention

43. Brand identity, Product recognition, General Public and Legislative Opinions

The SIT Subcommittee identified five priority areas from this list, for further consideration as system indicators. The priority areas are:

a. Market Penetration

b. Return on Investment (ROI)

c. Total System Investment

d. Customer Satisfaction

e. Customer Repeat Usage

The following presents the SIT’s recommendations regarding each of these areas along with the underlying background discussion

Market Penetration

Recommendation

The ability to measure the market penetration rate for both business and jobseeker customers is an important basic goal of the system. Furthermore, the development of market penetration measures would require partners to begin sharing customer information in a common database, which in turn would promote integration.

Market Penetration Rate – Business Customer = Business Customers

Total Number of Businesses

Market Penetration Rate – Jobseeker Customer = Jobseeker Customers

Local Civilian Labor Force

Discussion Points

Tompkins -- Business Customer base (business who has accessed our services/activities in the last 3 years) and total number of firms (business with as least one employee and pays into UI).
Sullivan -- Said this would be 10 employees or more and active employers (job fairs)
We can start to look at diversity or what it is we want to count in the universe. What data is available? We don't have VESID or other partner numbers: Employers need to be put in the database-and post their business contacts.
Total number of firms in that area
What firms do not want to participate in the public employment system?
Should we/can we count in the denominator those firms that are hiring or doing job development work, and then look at how many we served.
Which jobseeker customers would be counted in the denominator? Does this include all customers who have received some sort of employment related service by any of the partner agencies? For example, would it include TANF exhaustees who have been hired to work temporarily for the NYC Parks Department or participants of a Food Stamp Employment and Training program?
Is there the potential for localities to be selective in determining which jobseekers would be included so that they could maximize or minimize the rate? Which raises the question, “which is better, a high or a low Total System Investment Rate?” Could a high rate be better in some systems and not as good in others? Should the total system investment be linked to a specific outcome such as employment?
New hire data is available. Is this a subset of the first?
Are we utilizing resources correctly? When we look at hiring do we exclude those that just use the union halls?
In terms of LMAs or information what can we collect? DOL’s (DoES) definition of what can be collected is: personal visit; telephone conversation; job listing; job fairs?
We need to ensure that there is no duplication of the customer by multiple partners. Active businesses over what period? What can we reasonably access? The measures should be encouraging the use of a system wide collection of business services etc. Business wants a single point of contact. Therefore, we conclude that there needs to be a single data system. We need to follow the principles that we have established and then work towards the end goal of partners’ ability to report on these indicators. We need to ensure that all services: Core service, intensive and training services are done with an account manager.

A subset for Market Penetration: # of Firms (numerator)

# of Firms Hiring (denominator)

Problem - How to collect numerator data from areas with multiple Business/Employer databases? Is this a data collection issue or a definition issue? Integrated outcomes are the key and the desired goal. How do we define what an active business is? Time and services/activities: what constitutes the numerator?
First find out what businesses want to know about the system. The concept of market penetration is interesting. However presenting such a measure as a simple fraction with huge differences in the numerators and denominators is neither useful nor useable. A measure such as this would require the development of a regression formula that accounts for the weight of numerous variables within the labor force. Will this be useful to system users?
Some business partners have indicated that we should focus on: how many people were served and how much did it cost?
Services to business: define what you are going to count and not count. Some services are being provided but we are not getting credit.
Issue - Do we have the number of 14 and 15 year olds that have working papers? Do we include 14 & 15 year old

Return on Investment (ROI)

Recommendation

· Do not develop a system indicator for Return on Investment (ROI), as the methodology and underlying assumptions for computing ROI are highly subjective.

· Instead, focus on examining Total System Investment across the business and jobseeker customers.

Discussion Points

· Calculating Return on Investment (ROI) of a public program requires placing a monetary value on both the cost of the program and the benefits derived from the program. The methodology for computing these values is highly subjective, and gives rise to results that can vary considerably based on changes in the underlying assumptions about the costs and benefits attributable to system programs (e.g., reduced crime, reduced dependence on Family Assistance, Medicaid, Food Stamps, reduced UI duration, State tax revenues, economic multiplier effects attributable to employment income, etc.).

· Most ROI studies tend to overstate the benefits relative to the costs.

· On return on investment