NSB_Guidelines
REF Scholarship
Application Evaluation and Selection
Policies and Procedures
REF Scholarship
Application Evaluation and Selection
Policies and Procedures
- I. Main Steps in the Evaluation Process. 1
- II. Ranking System.. 3
- III. Evaluation Criteria. 5
- IV. Personal Interviews. 6
- V. Weak Points of the Application. 6
- VI. Academic Dishonesty. 7
I. Main Steps in the Evaluation Process
The evaluation process of the Scholarship applications varies slightly in RMUSP, LHP, RHSP, and RISP Program countries based upon the number of received applications and specificities of each program, the difference between the target group, and the format and time frames for rendering decisions.
Nevertheless, the evaluation and selection process has the following sequencing and general characteristics:
- Eligibility Check; this is performed by the Country Coordinators;
- Evaluation of the Online Application Forms (OAF) and of application materials; this is performed by the NSB members;
- Selection of the applicants; this is performed by the NSBs. This stage will contain an extra step (personal interviews), to be conducted with the selected freshmen applicants. In some countries all freshmen will be invited, whereas the number of freshmen is very large, a random selection of up to 60 applicants is made.
1. Eligibility Check – This step consists of a technical and administrative checkup, to verify if the application complies with Program’s requirements as specified in the application guidelines; this is conducted by the Country Coordinator.
The Eligibility Check does not judge the quality of submitted applications, but merely checks the information provided in the Online Application Form (OAF) and the submitted documents against the eligibility criteria (see here for RMUSP, here for LHP, here for RHSP and here for RISP). In this stage applications are either disqualified on the basis of non-compliance with the established eligibility criteria, or accepted in competition. Once an application is made eligible in the system it becomes available /visible on the evaluation platform, for the NSBs to evaluate it and assign their scores.
2. Evaluation of the OAF and of application materials – This is the second and most important qualitative step in the selection process. Each application is independently evaluated by the 3 NSBs. The cumulative evaluation score consists of points awarded for grade point average (GPA)which is calculated[1] automatically by the system , the quality of the written assignments such as the academic goals’ statement of purpose and the essay on Roma issues, the extracurricular activities, and the quality of reference letter(s), and for certain applicants also the interview. All above-mentioned criteria are weighted differently, as a percentage of the total score (see details below, under Section III).
Please note that the NSB evaluation platform will NOT show applicant’s Grade Point Average, but it will calculate it automatically for the final score. NSBs are expected to conduct evaluation based on the other criteria (i.e. A2-A5 and A6 where relevant), without taking into consideration the Grade Point Average.
When the online evaluation process is concluded by all 3 NSB members, the final score of each application is calculated as the simple average between the total scores of each NSB member. Besides the simple average, the system also calculates the standard deviation (SD) between the 3 scores of NSB members. The SD reveals the degree in which the NSB members assessed differently the same application, i.e. the level of disagreement between the three final NSB scores for the same candidate. The applications with the SD higher than 20% will be dealt with through email correspondence and /or discussed during the NSB selection meeting.
Example: An application is assessed with score 90 by first NSB Member, with score 80 by second NSB member, and with score 40 by third NSB member. In this case, the SD between the 3 scores in 26.5%. This reveals the high level of disagreement between the 3 scores (i.e. the third NSB member’s score is particularly low compared to the scores of first and second NSB members). This application shall be further addressed with Scholarship Program’s involvement, and with the NSBs, prior and during the Selection Day meeting. |
The NSB members are requested to rate each evaluated application by assigning scores from 0 to 100 to each evaluation criterion.
Attention! NSB members should NOT give scores summing up to a 0-100 points from all criteria, but to give a score on a 0-100 points scale to EACH criterion!
After doing so, the system will automatically calculate the total number of points, by using a weighted average formula. Once the evaluation process is concluded, the ranked list will constitute the basis for Selection Meeting discussions.
NSBs will conduct the evaluation based on information and documents submitted in the Online Application Form, and checked against the Application Guidelines requirements.
The selection meeting includes also the personal interviews, which will be conducted with those applicants that fall under the ‘to be interviewed’ category, as described in the application guidelines. The sixth evaluation criterion, entitled ‘[A6] Interview’ shall be used only for the invited for interview applicants. In case a candidate was not invited to an interview the [A6] criterion shall be left unchanged.
3. Selection of the Applicants – The NSB Selection Meetings in the Program countries, which include interviews with the new applicants, are organized for 1-1.5 days, depending on the number of applicants to be interviewed and the number of applications to be discussed.
Once the results of the individual evaluations by NSB members have been compiled, the evaluation list will include the categories of applicants as described below.
II. Ranking System
Each country and scheme has an assigned number of scholarships available.
Based on the ranking system, the evaluated applicants are grouped in three categories:
Category 1 – Accepted Conditionally applicants:
The list comprises applicants whose scores place them above the threshold (quota) number of available scholarships in the country and per scholarship scheme. Applicants on the ‘accepted conditionally’ list will become beneficiaries only if they submit by Program’s second deadline “enrollment” a valid and compliant enrolment certificate, as well as the cumulative benefits statement.
Category 2 – Applicants on the Waiting list:
The list comprises applicants whose scores place them below the threshold (quota) of available scholarships in the country and per scholarship scheme, but within a limit of 20% out of the quota. An applicant on the ‘waiting list’ could become a beneficiary (if the enrollment certificate and cumulative benefit statement requirement is fulfilled) and only if an applicant from the ‘accepted conditionally’ category fails to comply with the Program’s second deadline requirements.
Category 3 – Rejected applicants:
The list comprises those applicants whose scores places them below the category 2: ‘waiting list’.
Example:
If Romania has 600 eligible applicants and 300 available scholarships, there will be the following categories:
− Category 1 – Accepted Conditionally applicants; will be composed by the first 300 applicants based on the ranking;
− Category 2 – Waiting list; will be composed of the following 20% applicants, i.e. 60 applicants;
hence, there will be 360 applicants kept in the competition;
− Category 3 – Rejected applicants; the remaining 140 applicants.
During the NSB selection meeting discussions, the main focus will be on the applicants associated with SDs higher than 20%, in case their status could not be solved (by SP’s involvement) through email correspondence between NSBs, before the selection meeting. In the course of such SD-related email correspondence, the NSB members may maintain or alter their score on each concerned application.
Example:
Applicants |
NSB member 1 |
NSB member 2 |
NSB member 3 |
(i) Average score |
(ii) Standard Deviation |
(iii) For NSB Discussion |
A |
83 |
95 |
76 |
84 |
9.6 |
– |
B |
15 |
100 |
97 |
70 |
48.2 |
to NSB discussion |
C |
60 |
66 |
62 |
70.6 |
3 |
– |
D |
12 |
16 |
28 |
18.6 |
8.3 |
– |
E |
23 |
79 |
100 |
67.3 |
39.8 |
to NSB discussion |
Out of the five applications, two (B and E) will be discussed and ideally solved by the NSBs prior to selection
meeting. The others will be arranged according to the descending point scale (A, C, D). Of the latter, A will be on the top of the list and D at the bottom.
Based on the outcome of the NSBs’ decision concerning the high SD, the system re-adjusts the ranking of applicants based on the new scores.
If an individual NSB member revises his or her decision on the points assigned to a particular applicant, the Country Coordinator (CC) will note the change in the Selection Meeting Report. In parallel, the NSB member shall enter the alteration into the evaluation system, after which the list of applicants will be regrouped based on the new ranking. The discussion notes and alterations proposed by NSB members will be reflected in the Selection Meeting report, produced by the Country Coordinator and approved and signed by the NSB members in the meeting.
Following the conclusion of the selection process of the applications, the Program will generate:
- Personalized notification letters for the applicants on the accepted conditionally list and on the waiting list. The letters will contain the link to the submitted application for uploading the enrollment and the tuition fee certificates, as well as the cumulative benefits statement, and a note on the deadline for each;
- Personalized Rejection letters for the applicants who were (a) disqualified based on non-compliance with eligibility criteria, or (b) rejected based on selection criteria.
III. Evaluation Criteria
While entering the online evaluation platform, the NSB members will be able to inspect all country-related eligible applications (as they were submitted) and evaluate them individually. In order to view the eligible applications NSB members need to filter “Submitted” under Status and “Eligible” under Eligibility Status, from the online evaluation platform-GMS.
Each application is displayed with an attached NSB evaluation form. The evaluation form contains online evaluation option buttons, drop-down menus and check boxes for the evaluation criteria. The NSB members make selections and assign points from 0 to 100 for each evaluation criterion. It is recommended to make use of the entire scale for point assignment (i.e. not only round numbers, such as 10, 15, 20 or 80, but 11, 27, 53 or 67 as well), as differences among ratings will help positioning the applicant on the final ranking more clearly relative to other applicants.
There are two categories of applicants: new and renewal.
(1). Renewal applicants – students who received a REF scholarship in the previous academic year(s) and are applying to renew their scholarship for the same academic level (Bachelor, Master, Doctorate).
(2). New applicants, divided in the following two sub-categories:
(2a). Freshmen-to-be applicants – those who are going to study in the first year of a certain academic level (i.e. 1st year BA, 1st year MA, 1st year PhD).
– Students who received REF scholarship for a lower academic degree and are applying for a REF scholarship for a higher academic degree are also in this category (example: former REF SP beneficiary in BA, applying for 1st year MA).
(2b). Not-Freshmen-to-be applicants – those who are going to study in 2nd year or above and did not receive REF scholarship before within the same academic level.
Each application will be evaluated based on six criteria. The list of criteria, as well as the considerations for the evaluation and value of each criterion in relation to the total points can be found here for RMUSP, here for LHP, here for RHSP, and here for RISP. Please make sure you have the corresponding evaluation grid open when making the evaluation.
In line with the REF Scholarship Program policy and in order to maximize the impact of REF’s investment, the renewal applicants will have priority in the selection process. For this, the evaluation system assigns automatically “bonus points” to the renewal applicants, which weight as 20% in the evaluation grid. To make the electronic system use the relevant weighting system for the renewal applicants, NSB members are kindly asked to ignore the cell for the 6th evaluation criterion, since this will be completed automatically by the system depending on the category of applicants (e.g. renewals, with the bonus point, and freshmen with the total score from interview).
When the individual evaluations are completed and submitted by NSB members, the applications are ranked by the system, and Program administrators can review the summative results per each application/applicant.
IV. Personal Interviews
For the new applicants, personal interviews will be conducted during the selection process. The NSB members will complete evaluation for the interview part within the online evaluation platform, on the day of the interview, for evaluating the performance of each interviewee. The electronic evaluation of the interview opens in a separate popup window and contains 5 questions; NSB members will have to assign a score on 1-100 points scale for EACH of the respective five questions, and also write comments. When all interview questions are completed, the system will calculate automatically the overall score for the interview.
Personal interviews with applicants will help NSB members to reflect on:
● Level of commitment and drive of the applicant to pursue tertiary education;
● Ability to express academic goals and/or accomplishments clearly;
● Motivation to grow professionally and personally;
● Fluency of communication;
● Awareness of Roma issues.
The interview questions will be based on Appreciative Inquiry Technique and will help applicants reveal their strengths, comment on any underdeveloped aspects of their essays and focus on their accomplishments.
V. Weak Points of the Application
Besides the scores to be assigned to each evaluation criteria, the NSB members shall indicate the weak points in each application. Please note that the question about weak points is a mandatory part of the evaluation. In case an application is of good quality and therefore it is hard to assign any of the weak points, please choose the weak point in the list that states the following:
This application is generally good on all evaluation criteria. The only reason for which it could be rejected is the high competition and limited amount of funds that REF SP can allocate for scholarships in the respective country / scholarship scheme.
This will indicate to the Scholarship Program that the applicant is considered by the NSB member(s) as suitable for a scholarship, however, if rejected, it shall happen only because of the high competition.
Please note that in case an applicant has a relatively weak GPA score, the system will assign automatically an additional weak point about the non-competitive GPA score.
In case an applicant is rejected after the selection process, all weak points assigned to him/her will appear in the personalized Rejection Letter listed in form of bullet points; therefore while choosing the weak points, please keep in mind that this might end up in the Rejection Letter and serve as feedback for the applicant for his/her future applications. That is why it is important that chosen weak points are as relevant as possible to the application, otherwise this will constitute a ground for students to contest the rejection, which will entail more time needed for REF SP to deal with the rejection process.
VI. Academic Dishonesty
While conducting the evaluation of applications and the written assignments (Academic Goals’ Statement of Purpose, Essay on Roma issues), the National Selection Board members should take into account the Scholarship Program’s policy on academic dishonesty as detailed below.
In this regard, SP’s guidance on checking this matter is the following:
– whenever an NSB sees parts of written assignments which look very close to standard texts (such as from reports, articles, strategy documents, studies, researches, surveys, thesis, etc.) those parts should be copied and searched for in google, and as long as the results would indicate that the text is copied from other sources, this should be mentioned by the NSB in the comments, and reported to the Country Coordinator and the SP. The part(s) that appear to be plagiarized should be copied in the comments. While considering the score for that particular application, the NSB should consider those plagiarized parts of text as missing and assess the value only of the remaining text.
Each reported case will be analyzed by REF SP and a resolution is given, which entails a number of possible measures: disqualification, diminishing of the score, or assigning score 0 for that section (+ warning letter). Depending on the type of measure, further instructions will be sent out by SP to NSBs and or CC.
Plagiarism is the wrongful appropriation of another author’s language, thoughts, ideas, or expressions, and the representation of them as one’s own original work. REF SP considers plagiarism as academic dishonesty and a violation of Program’s values and principles. Therefore, applications in which plagiarism and /or self-plagiarism will be detected will be disqualified from the competition.
While completing the application, in particular the Academic Goals’ Statement of Purpose and the Essay on Roma Issues, the applicant is allowed to make a limited use of quoting, ensuring that the original source is properly mentioned and that the quote is not counted as part of the minimum requirement of words.
Self–plagiarism is the reuse of significant, identical, or nearly identical portions of past applications for REF SP schemes. Similarly with the situation of plagiarism, the self-plagiarism will lead to disqualification from the competition.
Please note that besides the checking conducted by REF SP staff and by NSBs, the Program may make use of anti-plagiarism software to detect instances of plagiarism in applications.
Furthermore, according to the REF SP’s Policies and Procedures, applicants found responsible for sending forged documents will not be considered further for REF SP and for any REF funded Programs. Any other instances of academic dishonesty related to REF SP application process and during the Grant Contract period (such as providing false /forged data and information, attempts to bribery, etc.) will also lead to disqualification from the Program.
Important:
NSB must be attentive to the issue of plagiarism and self-plagiarism.
For any detected plagiarized text, NSBs should provide relevant proof (link in case of a plagiarized text from an online source, and author and page number in case of a plagiarized text from an offline source- book, article, publication, etc.)
NSBs must clearly indicate the proofs of plagiarism within the relevant comment box in each application, by providing the proofs of plagiarism as mentioned above.