While most of the efforts focuses on implementation, we hardly ever pay attention to the long-term impact of an initiative outcomes. This is an issue of monitoring that is often active only during project life. Four years after the end of GEMM programme, we caught up with some of the actors to find out if the initiative had been sustainable and developed beyond the programme period… the outcomes are surprising… Still we have to learn from and use the results of a such impact monitoring to sustain change!
Comments (14)
Dear Abdelaziz,
Thank you for sharing this note, it is most interesting and encouraging. In particular, the replications of the project approach in Lebanon and Morocco seem very motivating for ETF's work.
However, I would like to underline that this kind of follow-up on the project five years after its end, however postiive it is, should not be a substitute for a comprehensive and systematic independent evaluation. Of course, the views of the implementers of the programme that you reached out for the note are crucial and very stimulating. But as implementer they have a vested interest in it, and it would be good to collect the views from other stakeholders, analyse the impact of the programme in terms of actual improvement of employability of the target population (most of the testimonies you reflect in the note, beyond the reference to the employment rate of 90% in the pilot project in Zarqa in Jordan, are employability promotion process-related, rather than employability outcome-related), opportunity costs, what obstacles remain to the achievement of the objectives of the project, or what factors made this project more successful than others, as you argue as project manager. All these elements would greatly contribute to the institutional learning curve.
I would also like to see some opinions on aspects of the project that, with the perspective of time, might be impoved: it may be a professional deformation as academic, but as a general rule any assessment of progress not containing any element of self-criticism seems suspicious to me; I had to analyze the evaluations carried out by implementers of almost 200 migrant support measures from an employment and skills perspective worldwide for the ETF's MISMES `project in 2014-2015 and I must say that this left me with a structural mistrust towards self-evaluations by international organizations and public administrations). I think that "Monitoring and assessment" withinh ETF, as one of the three services foreseen within the ETF’s 2027 Strategy, should ensure that those comprehensive and independent evaluations are carried out.
In any case, it is a great idea to reach out to former implementers and to gather their impressions on the GEMM project a few years after it was implemented. for the purposes of the ETF OpenSpace adn for those who do not know the project, it had been useful to add a brief project description at the beginnning of the note. Thanks for sharing!
GEMM was a great project and we at Transtec were happy to contribute to its success. I am very glad to read about these follow-up initiatives and sustainability of pilot projects beyond 2016. Thank you for sharing!
Thanks Asel for your comment and for your good support and collaboration during the implementation of this marvellous journey with our partner countries!
GEMM is an example that sustainability is possible but that it does not happen "by chance" but "by purpose". In the GEMM project design, the sustainability options were part of the project actions. This implied an active "search" and networking with national organisations, EU delegations, other donors, to prepare a sustainability plan. This could be e.g. embedding results into the national policies and VET reform programmes; scaling up pilot actions for a nation wide coverage through additional EU and donors funding; etc.
Another additional important aspect is the impact. GEMM was a fairly "small project" in terms of funding: approx. 3 m€. With this amount impact can only be ensured by building sustainable longer term actions (see my comment above) and by having a differentiated approach country by country to best fit the country needs and context. This was possible by a very targeted approach to the pilot projects components. And yes, the beauty of GEMM is that it was able to combine a differentiated country approach with a rich country peer learning and exchange approach. A lot to learn and thanks to Aziz for having reminded us of this successful experience
GEMM even becames an academic practice
Thanks Aziz for sharing this document, it is also a very good opportunity to take a moment to reflect and take stock also of how our work has followed up to GEMM in the region and how we have made use of the experience gathered.
From my side, I remember that there was a very important skills anticipation component. And the work which is currently done in Morocco, for instance, on the Observatoire des Branches (Sector Observatory) is a good example of it.
During the years, we have continued to stress that anticipating skills needs is absolutely fundamental to prepare change in the countries and support the transition toward the future. That was also one of the conclusions of our 2018 international conference on Skills for the Future: the need to strengthen the capacity to observe change in skills demand and take action accordingly. This has also being picked up prominently in our new ETF Strategy 2027, where the Skills Lab initiative will exactly have the function to provide more regular flow of information about changing skills needs in developing and transition countries.
Therefore, I believe that GEMM - among other things, also contributed to this more general reflection, as it gave us the confirmation that partnership and skills anticipation are fundamental topics that countries want to bring forward to improve their human capital development policies.
Indeed, skills anticipation was one important pillar of GEMM around which partnership is capital. We are in fact working on a proposal to support Morocco in skills anticipation at local level. Your support would be me more than welcome.
Please log in or sign up to comment.
Hi Ivan. I hope you are doing. Thanks for this deep and interesting reaction. My message was on purpose provocative and has at many sides. One is indeed like you said, the need of monitoring and comprehensive evaluation of the impact to be used for policy learning and adjustment, which we (ETF and most of IOs) are not doing in general. So the point here for us first (ETF) is to develop the "Learning" aspect of on the new strategic pillar "Monitoring and assessment" within the ETF’s 2027 Strategy. The second point is that sustainability needs to be foreseen in the design of the project and considered along the implementation. Without self promotion-), this has been the case for GEMM mainly through the link and involvement of the EC (EU Delegations in partner countries) and international institutions that have enlarged the initiative in most countries.
What we have done is a small initiative consisting on interviewing some practitioners in some GEMM participating countries and does not replace a real evaluation as you rightly said.
Anyway, it is a pleasure to hear from you and that for sharing your ideas and enriching these exchanges.
70
The ETF regularly commissions external evaluations for most of its major activities. These evaluations are published in the ETF website and are subject to public scrutiny and discussion.
https://www.etf.europa.eu/en/about/mission-planning/monitoring-evaluati…
These evaluations use a comprehensive approach and include Impact and sustainability as two of their criteria, as understood by the European Union and the DAC-OECD. They also systematically adopt a participative approach, capturing the perspective of the different stakeholders, including -but not only- those who participated or directly benefitted from the programme.
Finally, evaluations follow a formative purpose, with explicit intention to identify lessons learnt and recommend the ETF on improvement and learning.
For sure, the quality of evaluations, as it is for any activity in a centre of expertise, can always improve. I do not think, though that it can be stated that the ETF "is not doing in general monitoring and comprehensive evaluation of the impact to be used for policy learning and adjustment".
Thanks 68 for this clarification. As you know first these “Formal” Evaluations don’t cover all our initiatives and second and most important is the use of the evaluation outcomes. I really think that we don’t learn enough from our success and failure .... and this is the point here, now that we have one dedicated service to M&E in the new strategy 2027 (in addition to policy advice and knowledge Hub).
70
Sure. There is room to improve the learning potential of these evaluations. However I see it is useful to distinguish between M&E of our activities and projects and the policy monitoring of the country policies and system performance. In this respect the discussion about project impact is always in terms of contributions to the achievements and long-term results of policies where different actors, measures and inititiatives contribute.
Indeed, not easy to mesure the contribution to the impact. However, Monitoring and assessing regularly outputs and outcomes with a strong learning dimension, can help appreciating the contribution to the impact