Category Archives: Program Evaluation

Nonprofit Management, Performance measurement, Philanthropy, Program Evaluation

What can we learn from OTF giving history – Part 3

March 9, 2019 ready to think

ACTION AREA

As promised in the previous posts 1 and 2, I now turn to Action Area and related variables, in the goal of understanding OTF strategy better. Just to remind you OTF action areas are as the following:
Active People, Connected People, Inspired People, Promising Young People,
Prosperous People, and Green People. Those areas supposedly organize grant application under sub-umbrellas, and allow better comparison between applications and grant outcomes.

The first analysis that came to mind was to compare areas (across years and types of grants) in respect to the Total awarded amount and monthly dollars. Interestingly enough, there are significant differences (F***) in both the total amount received and in monthly support, as the following:

The chart above clearly shows a tendency to support in higher amounts for Active People projects against all other types. It also seems, however, that the most funded areas in general are Green, Promising, and Prosperous. Inspired, Connected, and Active are somehow clustered together in the second priority.

Moving forward, I analyzed the connection between Action Area and Type of grant. Here, as well, differences are prevalent:

40% of the Capital projects are in Active people area, and 24% are in Inspired.
Grow is more equal, but Promising Young get 25% of funding, seconded by Connected with 20%.
Seed has an indefinite preference for Connected (25%), and all other areas get varying amounts in the neighborhood of 13-18%, besides Active which gets only 9%.

Looking at Action Areas in general, focusing on type of funding, it seems that:

Active: 64% Capital, and 20% Seed
Connected: 49% Seed, and 28% Grow
Green: 53% Seed, 41% Grow
Inspired: 39% Capital, 36% Seed
Young: 45% Grow and 45% Seed
Prosperous: 45% Seed, 34% Grow

All in all, OTF strategy is certainly not equal in the pots of money across areas, and shows a clear tendency to prefer certain areas over others especially when it comes to specific types of funding. Those may be the results of specific tendencies in applications (i.e. Active applications tend to be focused on Capital requests rather than Seed; or Grow projects are more relevant in Young and Green). However, this analysis cannot account for those explanations, as OTF published only awarded grant data, and not full application requests.

GRANT RESULT*

Regardless of the Type of funding, I was very curious to check out the frequent/prevalent Grant Results. Those who are more frequent certainly fall under at least one of the following category (if not all): selection committee favourites, OTF team favourites, most frequent in requests, most frequent in agencies/organizations need, significant elements in projects growth, significant in sector-wide stability and growth, capture/promise change and opportunity. You may add more reasons as you wish, and I am happy to include them here for the benefit of my readers.

So, here are the top nine (afterwards come all 3% or less):

It is clear that the first and foremost concern is: social isolation. This result is significantly prevalent than others, and this simply means that if your project is connected to this result, it will be wise to choose it (use discretion and caution here, and choose it only if really connected to your project).
Second in line are: physical activities and culture/heritage. My thinking is that these two are highly connected to social inclusion in this way or another, so we can say safely, that an application about bringing people together whether if for sports, arts, crafts, event, festival, social program, get-together, etc. is definitely going to get significant and positive attention in your favour. If you look at the rest of the list, you will easily see other variations of physical activity, social inclusion, and you will see it narrows down to children and youth.

In this case, I can clearly say there is an evident pattern to prefer projects that are focused on Social Inclusion, Group Activities, and more specifically projects that target youth, children, and suggest programming around culture/heritage and sports. This is not to say that most of the money goes to these areas, but undoubtedly, those results are the most frequently funded ones.

This analysis also shows the preferred result for each top area:

Connected: People who are isolated have connections in their community (n=301, %=35)
Active: Infrastructure for unstructured and structured physical activities (n=205, %=34)
Inspired: Arts, culture and heritage have appropriate spaces (n=180, %=21)
Young: Children and youth who are facing barriers develop strong emotional and social skills (n=165, %=19)

*inluding Ontario150

Next, I analyzed the four top frequently funded areas, to see if funding dollars are also different. Strangely enough, there are real differences and trends in the total amount funded and monthly dollars.

Total highest $ go to Infrastructure (makes sense) and Children/youth (unclear why or how it’s so different than isolated people in general). Highest total monthly $ go to Infrastructure (makes sense) and another version of Infrastructure (makes sense). All in all, again, when you are making an application, if you include the social isolation component, you have significantly higher odds to win.

It should be noted that I looked into two other variables: Population and Age. However, majority of the grants were awarded for “general populations” without any certain age limit (children over 12 yrs and youth comprise of about 30% of total Age; people with disability 15% of total Population Served).

That’s it for this post, I hope you enjoyed and learned. On my next post, I will wrap up the discussion and focus on major findings in this analysis in the goal of identifying better how OTF strategy works, and how you may improve you application in the next round.

Nonprofit Management, Philanthropy, Program Evaluation

WHAT CAN WE LEARN FROM OTF GIVING HISTORY? PART 2

February 10, 2019 ready to think Leave a comment

In my previous post, I provided general information in respect to OTF grant strategy. This info sheds light on giving amounts, favorite grant types, long term strategy, and in general makes some order in thousands of raw data information. When trying to understand better how exactly the grants are divided geographically, I was dealing with technological challenges including pricey software and limited sharing capacity.

[long story short: started using Tableau, which is highly friendly however I could not share and was able to analyze on the first 3 letters on postal codes; then switched to PowerBI, which is significantly less friendly and shareable (although limited in time)].

All in all, I will present here maps that show giving dollars and postal codes, and clearly show where the money goes, and how much. Not surprisingly, population concentrations get more attention and funds (in general), but if you are interested in a specific location – those maps can give you highly effective visualization of the data. I was surprised to learn that maps can shape my understanding better than any other graphs that I dealt with so far; so even if you are experienced with data analysis for years, you may still be pleasantly surprised to be suddenly aware of the potential insights you enjoy while using in data mapping.

Back to business, the map below shows average giving per month based on the postal code.
The range is between $2,125 and $265,883 per month. Quite wide (I did not control for type of grant in this map). Due to technological constrains, I could not share the interactive map with you; so I captured the core, and left the rest of the province outside this image. However, we can easily see that GTA gets attention, and interestingly the periphery gets higher amounts (the dark blue on the right is K0K ($166k), and the semi-dark blue on the left are N0G ($87K) and N0H ($81K). The two small areas in the middle of the map are L4R in the north with $190K and L3Y in the middle with $178K.

Moving forward, I created an interactive map for total giving amount across the province. In this case, you can explore the area of interest and see how much in total was awarded per postal code.

Seed – $ per month

Diving into Seed grants over the years, it seems that there are actual differences between postal codes. At a general glance, the core of Toronto/York Region area does not get too much Seed if any, and the average dollars per months significantly vary. This raises the question whether organizations in this area are well established and therefore need less Seed funding, or maybe OTF strategy proiritizes other areas that are less central. I do not have an answer for this question (OTF do not publish unsuccessful application data), but you should take this into account if you are planing to apply for a Seed grant.

The map shows that the most funded is Toronto Biennial of Art (M6P) with over $35,000 per month. The least funded on a mothly basis is St. Vincent de Paul Society of Kingston (K7K) with just under $550 per month.
I cannot describe in words how exciting it is to play with the data over the map, but unfortunately I could not share it online.
Over-time analysis shows a trend of growth in average monthly funding: from $5400 per month in 2015/16 to $5550 per month in 2017/18.

(Average duration for Seed is 11 months).

On PowerBI it looks much less informative, and I could not find a better way to present the very same data. Anyways, here it is for you to interact with the data of interest.

Grow and Capital geographic distribution

Capital across the province (per month):

In general, again, it seems that the core of Toronto/York Region is less funded than other areas. This requires further investigation, as it is not explained by this data why the core is funded less than the secondary tier. Maybe additional variables such as populations and action areas will explain better.

Grow GTA (per month):

This map again raises more questions than answers. Why the core is not represented? Why there are areas that are heavily funded? Postal codes in the middle of the map: L0K, L0H, L0B, L4A, L7K (all around $18,000 per month); Toronto in the areas of M+.
No clear picture of funding strategy comes up from this map.

The only explanation I have now is that postal codes are more dense in more densely populated areas; and this may create a biased visual of funding distribution. However, this does not provide an answer to the question of monthly awards; and does not provide a good answer why over years some areas get zero attention.

At this stage, until I find a better solution, or I get positive feedback about PowerBI maps; I have no plans to produce interactive maps for Grow and Capital. Please write me if you think this is beneficial.
However, regardless of maps the below presents a summary of monthly dollars. In my opinion this is a good guide if you have future plans to apply.

Growth in monthly funding is prevalent and significant

When I tested Grow and Capital – growth in funding is clear.

Grow monthly avg. in 2015/16 – $10,500; 2018/19 – $11,900. (average duration is 33 months).

Capital monthly avg. in 2015/16 – $12,900; 2019/19 – $15,300. (average duration is 8.5 months).

In other words, the average grants you may want to apply for next round (assuming no significant changes in the process) are as the following:

Grow for three years, for about $400,000; or Capital for 8 months, for about $130,000. If judging from the theoretical amount you may get -Grow falls way behind the maximum of $750,000, and Capital obeys the defined boundaries of $150K.

By this I will end the blog post for maps. In the next posts I will focus on
Action Area, Grant Result, Demographics, and Funding Dollars.

Please feel free to share, comment, and contact me for insights and ideas.
Thank you for reading!

Performance measurement, Program Evaluation

On the credibility of credibility tools

January 19, 2019 ready to think Leave a comment

Most of us, who work in the line of evaluation, bear in mind and remind ourselves and others about validity and reliability of measurement / evaluation tools. When I design a study, I always think how to triangulate the collection of data, and use more than one system to measure the subject in question. Therefore, most of us will usually use several questions to measure the same indicator (and then conduct a reliability test); and ensure the test and indicators are actually measuring the topic we would like to learn about. However, I never put too much thinking about other tools and research instruments that are perform other types of measurements, i.e. Polygraph. What polygraphs are entitled to do, is to provide the researcher/authority with some information that in general is considered more credible than just another statement or testimony given by the participant. Much research was and is done on this regard, and it is widely known that polygraphs are not too credible or reliable tools to assess whether the participant tells the truth (a not so credible way to assess credibility!). This arises several questions:
(1) why is it s widely used, while known to be less reliable than the average person would expect this to be (not to mention experts)?
(2) why did humanity could not come up with a better, more reliable, more valid solution so far?
(3) what are the consequences of using such a tool on human rights, dignity, and justice?
(4) how can we improve the tool, or suggest a better tool, or at least suggest a tool to triangulate and validate polygraph findings?

It appears those questions are high priority these days, and there is a competition in the US, focusing on “Credibility Assessment Standardized Evaluation (CASE)“. This Prize Challenge offers five prizes to teams and individuals who will suggest fruitful tools to asses and standardize evaluation process for credibility tools. In their words: “The CASE Challenge is … to develop credibility assessment evaluation methods that can be used to objectively evaluate both existing and future credibility assessment techniques/technologies”.
Registration is open now, and those who will present winning solutions will be invited to Washington, DC in summer 2019. I am highly curios to learn what options we have to create a better, standardized evaluation, especially focusing on intended future behaviour. A reliable, standardized solution may be duplicated to other areas such as program evaluation in education and social services.

Nonprofit Management, Philanthropy, Program Evaluation

Grant Allocation and Management

January 5, 2019 ready to think Leave a comment

In small family foundations, the prevalent question is “how can we do best with our investment?”. Similarly to the question about fund investments in stock and other assets. However, the big question of how to impact the most with the funds, may be answered quite decently when taking several aspects into account:

(1) applicant information: what information is collected from applicant, and how much information is usable. In other words, the secret of data collection in the first stage is the lean nature of the collected data. Ensure the information requested is highly relevant for the decision making process.

(2) clear guidelines: when the foundation is getting too many irrelevant applications and letters, this raises the question of clarity and accuracy of guidelines for applications. Hence, ensure the application guidelines are clear, presented and accessible on the website, and state shortly and clearly who are the typical applicants to be considered. Also, provide a clear timeline for application rounds and decision announcements.

(3) decent evaluation: the most important, yet frequently neglected aspect, is evaluation and understanding of the contribution impact. It is easy to donate for a cause close to one’s heart. However, grant making is not only about giving, but also about ensuring the investment is impactful and fruitful. By this, the foundation needs to create simple rubrics or other evaluation tools to allow continuous and clear reporting (quarterly and annually). The evaluation tool should also provide the opportunity to compare between funded projects, to assist in decision making for future donations. Those evaluation tools do not have to be too sophisticated, they can be smart enough to compare and extract information based on the criteria on the guidelines, and most importantly – measure and evaluate the match and achievement of the foundation’s vision.

(4) organized decision making: when all the above are accomplished, the foundation’s management team and board are able to make decisions quite easily and effectively. Whether it is about deciding who are to be included in the next round of applicants (i.e. section 1 and 2 – guidelines and applicant information) or what was the impact of the invested funds over the year, and how the projects are doing in comparison (section 3). The management team arranges all the information for the board, and the board makes efficient, wise, evidence-based decisions to highly impact and use the funds in the most significant way to achieve the foundation’s vision.

For information about our services please click here.

Nonprofit Management, Performance measurement, Program Evaluation, Work plan - vision, goals, objectives

Planning Programming Budgeting System (PPBS)

January 1, 2018 ready to think

Guest Post by Ron Sommer

Introduction: reading my post about system thinking and program evaluation, Ron mentioned a close practice that incorporate both budget and planning. I am glad he agreed to write a guest post about the topic, and hope you will find it fascinating too.

Moreover, this is the first blog post for 2018! and opening the year with such a great topic is a good reason to celebrate (:

***

In ancient days, meaning the 1960’s, Planning, Programming, Budgeting Systems (PPBS) was considered an innovation in budgeting. PPBS was first introduced in the Defense Department in the USA in 1961 by Robert McNamara, and in all departments in 1965 until 1975. Though it failed to be widely adopted in government, PPBS is effective is less complex organizations such as NGO’s.

PPBS is an integrated management system that places emphasis on the use of analysis for program decision making. The purpose of PPBS is to provide management with a better analytical basis for making program decisions, and for putting such decisions into operation through an integration of the planning, programming and budget functions. Program decision making is a fundamental function of management. It involves making basic choices as to the direction of an organization’s effort and allocating resources accordingly. This function consists first of defining the objectives of the organization, then deciding on the measures that will be taken in pursuit of those goals, and finally putting the selected courses of action into effect.

Planning Programming Budgeting System (PPBS)

An organization can be viewed in a simplified way as carrying out its functions through five basic and sequential phases: (1) planning, (2) programming, (3) budgeting, (4) operations, and (5) evaluation.

Specification of Objectives – The objectives of the programs are to be specified in consistence with the long-term goals in quantitative terms as far as possible.
Systemic Analysis – The possible alternative projects to achieve the program objectives are analyzed in a systematic way with the use of cost-benefit and cost-effectiveness analysis.
Functional Classification – The budget is classified on a functional basis like functions, programs, projects and activities.
Organization – Budget formulation addresses the organizational structure, managerial and administrative procedures of the programs/projects/activities.
Evaluation – The mechanism for evaluation of performance on the basis of financial and physical performances to monitor, and take corrective actions, if necessary.

Each of these phases consists of a distinct but related function in the overall conduct of the organization’s affairs.

Planning is an analytical activity carried out to aid in the selection of the organizations objectives and then to examine courses of action that could be taken in the pursuit of the objectives. Planning, in effect, poses the question of whether some particular course of action would contribute more to the attainment of the organization’s goal than its various alternatives.
Programming is the function that converts plans into a specific action schedule for the organization. Programming consists of developing detailed resource requirements and the actions needed to implement plans.
Budgeting is the activity concerned with the preparation and justification of the organization’s annual budget. The function of budgeting is to secure sufficient funds to put the program into operation.
Operations consists of the actual carrying out of the organization’s programs. Preparing for operations is the object of all the other phases.
Evaluation is the function that evaluates the worth of operating programs. Through program evaluation the worth of programs in attaining goals is measured and appraised. The result of evaluations is used to modify current operations, if indicated, or in planning future programs.

PPBS provides an opportunity for identifying the program alternatives which offer the biggest pay-off in achieving communal objections, or require lower costs, and these can be singled out for priority attention by planning groups.

Program Evaluation

DIY: The Crafting Job of Program Evaluation

February 28, 2017 ready to think Leave a comment

Professionals involved in program evaluation projects occasionally find that their understanding of how the work will be used and adapted by the organization does not necessarily parallel their clients’ perception of the project.
The DIY Method, a unique tool I developed, proves to make a vast difference in both collaborative work and the process of organizational performance, changes of policy, and program evaluation capabilities. The DIY Method begins at building capacity. We aim to include all managers and stakeholders in the process of program evaluation to better find common ground and agree on the initial need for the evaluations. Secondly, we seek to clarify the research question and discover the gaps in information about the program or initiative. It is vital to recognize what is unknown and requires further investigation. Thirdly, assuming that our research question is well defined, we then focus on constructing research tools for program managers or officers to use for data collection. These tools contain basic statistics and cross-tabulations to assist in day-to-day management. They can either be simple Excel based software, or built into the CRM or preexisting organizational database. It is decided by the clients which method works best and allows for smooth and beneficial program evaluation. This stage may take some time as there is a need to adjust, apply, and modify the tools to contribute with managerial decisions and facilitate formative program evaluation. Our professionals are there to help create a beneficial environment and assist with program evaluation tool implementation.

When all is working and in place, our consultants analyze the data collected for the quarter or year, and articulate a research report to summarize the results, data, and trends. A significant portion of our time is dedicated to writing practical recommendations, supplying a beneficial use for evaluations, and assisting with continuous work to facilitate a smart and successful atmosphere for decision making and operations.

If this method of work interests you and you are curious to learn more, contact me today for a FREE initial consultation!

Nonprofit Management, Program Evaluation, Top Posts

Regression to the Mean and its Implications on Program Evaluation

August 19, 2016 ready to think

Background and Problem Identification

Most of the social / educational program are evaluated this way or another, and on this post I would like to focus on repeated measures of the same group of participants or individuals, as opposed to different groups comparisons or tests.

In many occasions we want to learn what is the impact of an intervention on attitudes, perceptions, and behavior; and by this we want to isolate the impact of the specific intervention, hence the program, and see how it changed the attitudes, perceptions, behavior, or a specific situation; in order to infer whether the intervention was effective or not.

Many of us will conduct t-test or repeated measures test. Another common way to investigate those questions is using a linear regression model; and by this try to predict the change on our dependent variable by a series of controlled variables. However, here comes the “catch” –

“Regression to the mean (RTM) is a statistical phenomenon that can make natural variation in repeated data look like real change. It happens when unusually large or small measurements tend to be followed by measurements that are closer to the mean.”
( Barnett et al., 2005)

The problem (RTM) may occur whether we measure an individual or a group, due to the random error (within-subject variance and between-subject variance).

A similar problem is identified as “a standard error of measurement (SEM), which refers to the standard deviation of an individual’s observed scores from repeated administrations of a test (or parallel forms of a test) under identical conditions”
(Koizumi et al., 2015)

The problem: variations in data sometimes DO NOT reflect a real change, but a correction of a previous random error.

In other words, we jump too fast to define a correlation as a causation, without checking carefully it really is!

Indeed, research conducted to investigate these measurement errors in social implications shows that many changes are accounted for RTM or SEM, and do not reflect a real change (Marsden amd Torgerson, 2012; Koizumi et al., 2015).

Solutions and Food for Thoughts:

Be careful when you aim to predict something. Do not assume a vacuum. On the contrary, plan the study cautiously and take into account alternative explanations, and different routes for interpretation. In fact, there is some good advise on how to reduce the chance your study’s results will be affected by natural errors such as RTM.

Research Design:

assign participant randomly for all groups
make sure groups are the same size
always include a control group
control for alternative variables
use tools with high reliability
control for background variables and context

Data Collection and Analysis:

conduct more than one pretest
collect two or more baseline data
control for baseline average / st. dev. by adding the group mean to the equation (either on regression or Ancova)

(Koizumi et al., 2015; Bonate, 2000; Marsden and Torgerson, 2012)

Implications on Program Evaluation

Many social and education program seek to change an attitude or perception, and assist participants in gaining knowledge of certain areas (such as financial literacy or second language).

Evaluation for these program usually focuses on perception measurement using a before-after design. Most of the time, RTS is not taken into account, and therefore interpretation of program impact may be wrong. Needless to say, designs without a “before” measurement worth NOTHING in terms of explaining program impact or change. In addition, there is a second aspect to emphasize which is the presence of a control group. Very often it is very difficult to compose a group of participants just for the sake of evaluation; however you should take into account that if you do not do it, you will never be able to correctly assess neither a baseline nor a change in your group of study.

In short: be cautious, plan and conduct evaluation carefully, when bearing in mind that a change in attitudes, perception, behaviour or knowledge, can be explained by a variety of explanations, that may be slightly different than the intervention you evaluate.

***

Subscribe if you liked (:

…and feel free to contact me regarding program evaluation consulting projects

***

References:

Barnett, A.G., Van der Pols, L., Dobson, A. (2005). Regression to the mean: what it is and how to deal with it. Int. J. Epidemiol. , 34 (1):215-220.

Bonate, P. L. (2000). Analysis of pretest–posttest designs. Boca Raton, FL.

Guisasola, J., Solbes, J., José-Ignacio, B., Maite, M., Antonio, M. (2009). Students’ Understanding of the Special Theory of Relativity and Design for a Guided Visit to a Science Museum. In: International Journal of Science Education 31(15), 2085-2104

Koizumi, R., In’nami, Y., Azuma, J., Asano, K., Agawa, T., Eberl, D. (2015). Assessing L2 proficiency growth: Considering regression to the mean and the standard error of difference. Shiken, 19(1).

Marsden, E., Torgerson, C. J. (2012). Single group, pre- and post-test research designs: Some methodological concerns. Oxford Review of Education, 38, 583–616.

Performance measurement, Program Evaluation, Top Posts, Work plan - vision, goals, objectives

4 Questions to Answer Yourself Before You Measure Outcomes

October 24, 2015 ready to think Leave a comment

Hello friends, I am now writing for over 6 months, and every time I publish something I feel that I somehow helped the world (: So thank you for reading my comments.

Today I’d like to focus on a very sensitive issue – outcomes measurement. Yeah! you need to measure. If you make money out of your activity – you may want to know what was the impact in order to maintain and retain clients; If you do nonprofit you may want to know what was the impact of your services, and how you can grow.

I will make my points simple and clear:

1. Feelings are not true. They are misleading. Never trust your senses or impressions. Measurement does not work this way, and this is why you want to employ it. Facts are usually different than what we tend to think… a smart women quoted me a very strong message 10 years ago:”without data you are just another person with an opinion”. This is still so true.

2. Your boss/es are not interested in statistics and data, they just want the work to be impactful. Really?! how are you going to know it? Write down your goals, translate into objectives, and have these objectives measured. Be the one who brings rational and data to the table, be the one to help the organization grow.

3. You must have some sort of education with regard to measurement and evaluation. If you do not have, you are having very high possibilities to be mistaken. The most problematic trajectories are: questionnaires design and data analysis. I have seen thousands of questionnaires in my life so far, almost each one of them contained a critical mistake, which turns the whole business to be useless. So, bear in mind, neither your MBA nor MSW qualified you to write questionnaires and interpret data. Use professional advice.

4. What are you planning to do with the data? make sure there is a good reason to collect data and evaluate. Once you have the data, make the best out of it. Translate it into strategic steps and apply it. Then, recheck your performance and adjust accordingly.

Thank you for thinking and reading! On the next post I will share an example for a measurable impact questionnaire I created, for free use. Please stay tuned and subscribe (:

Performance measurement, Program Evaluation

Not-For-Profit BUT Yes-For-What: Reviewing The Solution Continues… (part 2)

March 12, 2015 ready to think

Ahhmmm…. got to tell you I am excited.
In the past weeks, I have had the opportunity to meet many professionals virtually, especially thanks to our lovely brother LinkedIn. I found myself in a process of learning, that I missed so much! well, enough with the introductions, let’s start thinking (-:

Just a short summary to get you to the point I am in now, in case that you did not read (you missed!) or do not remember (that’s fine (: )…
My first post on this issue dealt with the hard question – what are nonprofits for…and the essential need to connect doing to impact. My second post in this regard dealt with impact measurement, and my suggestion for three elements in metrics: Usefulness, Friendliness, and standardization.
I have got lots of feedback, and it led me to a cross-way:
One group of people maintained the operational aspect, as the main weakness. The other group agreed on the need of impact measurement, however everyone is kinda not sure what is exactly the thing needed to complete the mission.

The Operations Aspect

I would like to take it one step ahead, and correct me if I am wrong, the operations-minded people are usually dealing with something we can touch, or at least can see impact simply just by looking what the organization does – water, agriculture, farms, vaccines, you name it. The impact measurement in this type of organizations is a short and sweet ROI analysis. Efficiency, effectiveness, benchmark, and performance measurement are relatively easy to conduct, as well as goals and objectives setting.
Another important suggestion was to collaborate and cooperate, and even amalgamate organizations that do the same/similar work. I agree with this way of thinking but would like to ask you managers – will you work on your ego and let another organization to work with you or instead of you? Think about it. If it works – there may be great, extremely successful models of business supply chain management, which employ this attitude – more for less – by a chain of organizations. Phrase it like this:
The bottom-line is the VALUE profit. As long as there are organizations which may do the same, or even better than your organization – the real impact will be achieved by working together, and collaborate. It also saves you money, and reduces costs!
I know it is the hard part of managers’ ego, but probably one of the realistic ways to save the sector alive in terms of impact.
In short, the question to ask yourself is this – What is your organization’s unique selling proposition? What is your competitive advantage on others? What is the special value your organization creates? You supposed to have a very good answer to this question.
An additional note in this regard, is Collective Impact. Someone referred me to the collective impact website. I researched it, and browsed the web, and I must compliment them for doing the first operations step – creating coalitions and collaborations in order to increase the impact. However, this is only the first necessary step. The sector is getting shrunk, and will continue this trend, therefore there is a vital need to amalgamate or eliminate ineffective programs, not just collaborate.

The Social Value Impact Aspect

The problem of impact comes to life again when we deal with the social hot potato, and you know what?! I am dealing with it!
There is certainly a broad agreement on the need, although I must admit that I am still shocked to see huge foundations refuse to measure themselves (ego and power issues??)… But let’s put it aside. Just another small reference – social enterprises are a relatively small part of the nonprofit sector. I will never call them the “forth sector”, because they are not. I will never agree that they act differently than the third sector, they are value-driven, and this is the crucial aspect. They DO NOT care about money more than value, and DO NOT care about profit and value the same way. Therefore they DO NOT have double bottom-line, but one, and the last is very similar to the pure nonprofits. I can count few real social enterprises, but they will be again the ones we can touch – bakery, restaurant, agriculture, cafe, and the like.
Show me one social services enterprise… it ain’t exist, because it is impossible, and here comes the social impact measurement to help us.

So, based on my criteria – usefulness, friendliness, and standardization, and since I got no criticism on that (-: , let’s start.

I did not define Standardization last time, so here it is: in my opinion is needs to be simple. The metrics should include up to 4 core elements, which will be relevant across the sector. By this, there will be an option to compare between one organization to another. There is an option to add as many as indicators you like and want, and it won’t harm the metrics, but will give your organization the specific information you are looking for.

So, this is my review, happened somehow to be very small and narrowed…
I rank tools in 1-3 scale. 1=low, 2=medium, 3=high. Hence, the highest total score is 9.
I try to keep it as simple as possible, so do not rank 1-10 or something like that.

An important note! If your organization does not have a work plan which included vision, goals, and objectives, you cannot employ social impact measurement at this time. You MUST define the above in advance. Do not know how to do it? Drop in my post on setting goals, and keep up the good work!

SROI
I love this measure, however and in short, this does not apply in many social services and education nonprofits. If you are dealing with employment or any other outcomes which involve money, this may be the measure for you.
Usefulness (1): Applies to a narrow type of organizations (usually employment services, financial assistance, micro finance, and similar)
Friendliness (1) If you do not learn it thoroughly, and gain lots of knowledge – you probably won’t be able to conduct a reliable SROI analysis
Standardization (2) This part gets high score, because lots of research has been done, however there is no option to apply it broadly enough.
Total score: 4/9, 44%/100%

GRI / IRIS
I like the business attitude. This metrics will not save your life, but definitely will give you a way to benchmark your organization. This tool is used by many for-profits in order to monitor their performance, so I would rank it as the following:
Usefulness (1): it gets a low score here, because I am not sure how it is going to help many organizations in their day-to-day management in terms of measuring social outcomes. However some organizations might fall under the suggested social objectives, so I suggest to check it out.
Friendliness (2): the tool seems to be highly recommended and highly used by a variety of organizations. It does not gain the 3 points, because it does not fit every organization.
Standardization (2): you win the entire pot here. The tool is absolutely standardized, and you may feel free to compare your organizational performance to similar organizations in the industry. It is a huge advantage. I ranked it 2, because it does not apply in every field.
Total score: 5/9, 55%/100%

Social Impact Bonds (SIB)
I like the idea of social finance, because it makes much sense. It really builds a reputation for impact investing. In short, the system is designed to invest money in social projects, in order to PREVENT problems from reoccurring in the future (such as second-generation issues, recidivism, unemployment in specific sectors, etc.). The model briefly works by this: funding is given > intervention is made > evaluation of outcomes is conducted > in case of success (i.e. less recidivism, more employment) the government returns money to the investors. Even though I like the idea, I have no clue regarding the metrics and indicators they are using in order to evaluate social programs… it does not seem standardized or friendly, but it is just my outsider opinion.
Moreover, in my opinion, the social impact bonds model seems to fit to a narrow type of outcomes, kinda similar to SROI.
Total score: Unknown!

My list is much shorter than expected. I reviewed over 10 tools and methodologies, but did not like them at all, so why to mention them?

With that said, when it comes to social services and education, and other soft, hard-to-touch outcomes, the measures and indicators become useless. No usefulness, no friendliness, no standardization. Nothing helps. Therefore we must agree that there is another way to measure, and you know what?! it is not the kind, the gentle one… it is about achieving your objectives, measuring your VALUE. It is that simple.
Your objectives include “improvement of students’ grades”?
Show the improvement, between the beginning of the year and the end.
Your objectives include “women’s empowerment”?
Define what empowerment is, let’s say, they will be more responsible for their day-to-day tasks.
Show they have changed their behavior/attitudes.

Do not use excuses like “they are happy”, “their self esteem is higher”, “I feel it”… these are NOT your objectives, and therefore not the social value you wanted to create.

You will probably need a professional to help you with evaluation, it is impossible to do it alone, if you are not having a certain set of skills and knowledge (p.s. – yes, you may contact me if you need help with evaluations matters… (-: )

My next post in this topic will summarize my insights, and suggest a short-clear way to evaluate social impact; and yes – I am open for your comments and knowledge – just contact me and let me know.

I will be happy to get your comments, likes and shares. In the meantime… live beautifully (-:

Management, Nonprofit Management, Program Evaluation

They completely respect knowledge, and you?!

February 25, 2015 ready to think

Warning! the content may make some of you feel uncomfortable, and full of thoughts. I apologize in advance, and yet write about it.

Got to start with a short story… why? because this is the trigger for this very blast.
As you have already noticed, I do not write this blog for a long time, nevertheless I spent almost a decade (and counting) in management consulting in all its glory… and have a backlog of crazy stuff I want to share with you…

Once upon a time there was a large organization. Actually there were several of them, because it is going too sound you too familiar. The senior executives in this organization are the world’s smartest people, only God is smarter than them. The one and only wisdom is in their hands, and if they say so and so, you gotta say only one word: Yes.
The smartest people sometimes hold two nice shiny letters prior to their first name, yes you guess correctly – DR. Sometimes it takes the shape “PhD” afterward their last name. In any case they absolutely completed their third degree. I do honour each one of them. I have not completed mine yet.

So the smartest people were sitting in the Board of Directors and before letting anyone say a word in regard to an impact evaluation, they refused it out of hand, case closed. Why? because they do not need to know more, they can design it – just 2-3 questions, what’s the big deal??
In a different occasion they just ignored some suggestions for improvements, why? because they are the ones to know, not anyone else.

The story is the introduction for my little lecture below… ready to think?! I start… (-:
Smart people are smart and knowledgeable, but they certainly do not know everything. Assuming that their PhD is in Physics, they must know better than others in this field. If their PhD is in Nonprofit Management, they must be knowledgeable about the sector, but they are surely not experts in methodology, or business management, unless they have additional experience and education.
…and now I get to the main point: only methodologists are methodologists. Yes, every other academic has conducted research to some degree in some point of their academic career. However! that does not make you an expert in research methods. If you are a good manager, I believe you have great skills, but please do not assume you know to write a survey or design measurement tools. Mmm… no, you do not! You do not know how to define your question, gather data properly, you do not know how to create metrics, you doubtfully know how to analyze data, and utterly do not know to interpret it. If you want to know how to do the above – gain a set of skills and earn some experience.
I may generalize and say that my message above is right for every field, from business management to mathematics, and the bottom line is – respect the knowledge – it may help you more than you imagine! You know a lot, but be careful and aware of your boundaries. If something does not fall under your expertise, you better ask for professional help, or at least respect the advice given by people who hold education and experience in that area.
I also suggest to look around, you may notice extremely successful organizations that use experts in many aspects, there must be a reason to do so… (-:

By the way, I was really searching a movie for this post, because I hate thinking of myself as someone who is just teasing and criticizing, and not giving something relaxing for a change (-:
So, three birds in one stone, “The Beautiful Mind” : I love Russell Crowe, I truly like the movie, and I also find it inspiring and conveying the message. What is the message in my opinion? Short and sweet: Keep calm and respect experts (even when you think they are nuts!)

Enjoy!