impact of online assessment and online curriculum

Case Study Impact that the introduction of online assessment and online curriculum has had on the attainment and assessment of Year 9 students at the Academy of St. Francis of Assisi.

The Background

The Academy of St. Francis of Assisi is a city academy that was opened in September 2005. It is located in the Kensington area of Liverpool and has a population of 650 at present and will be at full capacity of 900 students by September 2009. The sponsors of the Academy are the Anglican Diocese of Liverpool and the Roman Catholic Archdiocese of Liverpool.

The school has an environmental specialism. This means that sustainability and the environment are key focuses across the curriculum.

The area has been awarded “Category 1” funding by the European Union, indicating that it is in the bottom 1% of cities for social deprivation and poverty. The unemployment rate in the surrounding area is 22%. The schools population reflects this, with over 52%[1] of students entitled to free a school meal (which is 3 times the national average) and 42% [2]students diagnosed with some kind of special educational needs.

The ICT in the new school is good with 3 networked ICT suites and further sets of laptops meaning that ICT is more available than in the previous donor school.

The teaching of ICT was when the school opened was based closely on the QCA sample units. These units were very literacy intensive and meant that students in most classes found the curriculum difficult to access. When the curriculum was suitable, progress was limited and slow due to student’s literacy issues. This meant that serious questions were asked as to whether the current scheme was fit for purpose. Due to the lack of consistency in the work being provided by students, assessment also proved to be an issue, with National Curriculum levels proving very difficult to award accurately as students where not demonstrating evidence of all criteria consistently- meaning that targets for pupil attainment where not being met.

This was identified as a particular problem at KS3 as the department’s results at KS4 had been consistently good, with students displaying a positive attitude towards the subject.

The Project

It was decided that a new approach was needed. This solution needed to address two main areas:
A supported curriculum that allowed students to demonstrate higher level skills and minimise the effect of low literacy levels.
An improved method of assessment. One that not only provided accurate evidence for summative assessment, but also one that would provide feedback to students and allow them to personalise their learning- very important under the governments “Every Child Matters” agenda.

We examined several paper based and ICT based solutions, however, most of these were rejected on the basis of expense or that they required a high level of literacy in order to access them which made them inflexible and difficult for staff members to differentiate for lower attaining students.

We finally hit on a two pronged approach that was, in the first instance, ICT based. This was especially appropriate because of the level of technology available to teachers and students during lessons, but also because the paperless nature of the proposed solution fulfilled the environmental ethos of the school.

The first prong was an online scheme of work which was adapted from one created for Notre Dame High School, Norwich. The environmental “Global ICT” ethos was especially suitable. The resources were web based and structured around the QCA KS3 strategy. The modular, chunked approach allowed colleagues to differentiate tasks more easily and I will examine this more in the next section.

The second prong was web based assessment. This was done using a Joint Assessment System created by ISIS software. This was a systematic approach to assessment, based on level descriptors taken directly from the national curriculum. This not only allowed teachers easy access to student work, but also allowed students to record there opinions of the work they had completed and receive instant feedback- again this will be examined in the next section.

The Theory behind the project.

Global ICT was chosen because the instructional design was quite flexible. The initial structure followed a traditional teaching method with a highly structured hierarchy of units, subdivided into lessons and lessons subdivided into individual tasks- of differing states and status.

The natures of these tasks are varied as they examine different skills. They build upon each other and as such are based in a constructivist approach as learning is broken down into bite sized chunks for the learner to digest.

The advantage of this web based system when compared to a traditional system is that all units are accessible to all students at all times. This means that if students wish to learn in a different sequence of events and construct their knowledge in a more meaningful way to them, then they are free to do so! This is in line with Prensky who theorised that modern “digital native” students have different thought processes to digital immigrants and rather than a linear learning journey, their experience is likely to be more parallel.[3]

The use of a core scheme of work also allowed the department to establish shared values and goals. The consistent approach allowed us to establish a “community of practice”[4], where there are shared approaches and goals. This has also allowed the department to view the scheme of work as a more organic item which is seen as the starting point rather than the finish. Colleagues have the time and confidence now to differentiate resources for groups in order to make things more suitable rather than having to create resources from scratch.

Importantly, the constructivist approach does not mean that students are not given the opportunity to go and apply the learning that has taken place. Instead, applying learning and allowing students to reflect on it gives the opportunity for metacognition and actually allows for higher level deeper learning to take place.

Some people would see this approach as “blended” rather than “e” learning. However, I see this as being largely irrelevant as in this particular context students need the support they get from teacher contact; the learning is not designed to be delivered remotely.

The assessment system allows teachers and learners to be involved in an “assessment for learning” approach. The cornerstone of this is that students are able to state which assessment criteria they feel they have met. They are able to upload evidence (via attachments) onto the system and then the teacher is able to draw upon this, their observations of the students’ attainment in class and the students own judgements when assessing the work.

The unique selling point of this system is that when a students completes their assessment, automated feedback is given by the system stating what the students needs to do in order to get to the next level. This allows a more independent approach for the student and means that meaningful, understandable feedback is given instantly- thus meaning that students are more likely to take action on that feedback as the learning is “fresh”. This also addresses a major problem of online systems, in that meaningful feedback at a time when it is needed is difficult to achieve as the tutor is not always available. Whilst the feedback given by the system is generic it is better than nothing.

Analysis

The first limitation to state when trying to analyse the effects that these changes have had is that we are dealing with immediate effects as the program has only been in place for nine months. This means that data sets are largely incomplete so results for current year 9 are current results and will change. However within this constraint it is still possible to draw comparisons.

In order to compare the initial impact of the systems I have taken two controlled samples of year 9 boys. Both sets are set 2 out of 5, both sets have been taught largely in all male classes and both sets have had the same teacher. I did this to try and standardise the sets as much as possible and while the results are by no means statistically significant, they are still extremely interesting:


Considering that the 2007 data still has 3 months to run, it can be seen that there has been a significant impact on the percentage of students gaining level 5 or above, and by the time the 2007 cohort has completed year 9 they will have seen an increase in the number of level 6’s.

The “Global ICT” online scheme of work was free. However, there has been significant time taken to rewrite some of the units that required some work and also management of the system on the school’s intranet. However, these are not direct costs and it is therefore difficult to put a financial value on them.

The school already had an intranet, although it was not populated- the adoption of this system has proved the catalyst for the development of a whole school intranet and has provided valuable understanding for the further development of an “e” learning environment.

The Joint Assessment System, initially cost £150 per year for a subject license. This has since increased to £250 per year. In my opinion, this has been extremely cost effective as the time savings that the new technology provides teachers far outweigh this. Again there is an indirect time cost and the system does take time to manage effectively- although this has largely been done by a teaching assistant employed within the department.

The overall coherence as an educational experience is an extremely positive one. This is because both have been based around the National Curriculum for ICT. They are both looking at achieving the same thing- raising student attainment. We have been able to nest opportunities to use the Joint Assessment System within “Global ICT”. Students complete their own assessments of their work at regular points through the scheme, from their perspective the two systems overlap seamlessly, creating a positive platform for learning to take place.

What is difficult is to decide what part of these two systems has been more effective. Is it that the structure that these systems provide allows the student a more secure environment in which to succeed? Is it that the more focused assessment system enables staff members to identify where students have met certain criteria and thus this has helped to raise levels (although actual student attainment remains unmoved)? Is it that the student self assessment clarifies assessment criteria and thus allows the student a better opportunity to show what they can do.

Is it that the “e” learning approach is more appropriate to students and this has allowed them to learn more? Does the breakdown into tasks and the supporting materials foster a different attitude towards learning- especially with lower sets?

Is it that the combination of the two systems results in a sounder approach to teaching and learning and displays a firmer, more consistent, transferable pedagogy that can be shared by all members of the department?
Conclusions

In order to help answer the previous questions and draw my conclusions I will first try to evaluate the solution that we have installed.

There are many positives that have been experienced by the department.
Ease of use in the subject. ICT is subject where there are obvious advantages for using an ICT based solution for dealing with teaching learning and assessment. Both systems are especially easy to use and integrate well with the use of other technology in the classroom, such as projectors and interactive whiteboards- enabling colleagues and students to maximise the educational value of these technologies.
The system is more easily understood by students. Both in terms of assessment but more fundamentally in the lesson themselves. The question “I don’t know what to do?” which is quite a regular one is much easier to answer and students who have listened first time are able to be increasingly independent in their learning. The assessment which traditionally has been the hardest thing for students to understand is much clearer. It is interesting to see what the students’ opinions are as to what are the considered easier topics and the more difficult ones. Digital natives do not always have the same opinions as the digital immigrants.
Students have made more progress. This is largely due to the provision of task templates. This means that students with literacy issues are supported and are able to demonstrate skills rather than demonstrate literacy issues. This has also helped to boost student self esteem and attitude towards the subject.
The system enables the department to meet the challenges of the “Every Child Matters” agenda in ICT. This is because it caters for a wide range of learning styles, especially the inclusion of some online gaming resources from www.Teach-ICT.com. This allows students to construct knowledge in their own way and this has been observed in lessons where student conversations demonstrate recall of complex ideas and concepts.
Assessment is undoubtedly more accurate. This is because all assessment decisions are made on the criteria. This ensures that students are assessed on their own merits and not on those of the students around them. We are able to quantify grades to the students and justify why they are what they are. What is 8 out of 10? What is a C+?
The general quality of teaching and learning has improved and the delivery is much more consistent. This has been seen in the observations across the department with all colleagues being graded at 2 or higher.

However, there some disadvantages to such an approach:
A single department approach to assessment does make whole school consistency more difficult. The problem with this particular system is that it is difficult, even with good ICT facilities, to accommodate the amount of computer access that would be required for the students to record their judgements (I will temper this by adding that departments could introduce more criteria based assessment.)
The creation of criteria, input of information into the system, and management of the system once up and running did take time. The fact that the Joint Assessment System does not talk readily to the school management information system also meant that certain tasks were duplicated, or not all the functionality of the system could be used efficiently.
The two systems are victims of their own design. They are largely focused on levels 4 to 7; this means that our lowest ability students have found it difficult to access the system.
The interface is very literacy intensive and so students in our year 7 lowest sets have again found certain units difficult to access. However, in year 8 the lowest sets have made excellent progress with targeted support.
JAS has not been accessible from home and this means that assessment judgements have to be made in school rather than at home. There are advanced plans for a virtual learning environment to be installed and this could be addressed- although still to be confirmed.
The adoption of the virtual learning environment and developments within the management information system could make the entire assessment system obsolete within two years.
Although the assessment system has been used effectively in the department, it has not always been used consistently, meaning that initially there are “holes” in some parts of the data. There have also been inconsistencies as to how students have “uploaded” work and whether or not they have then taken notice of staff comments on the system.

In conclusion, the impact from the subject point of view has been extremely positive. The positives listed above are all major wins for us as a department and the initial attainment data from year 9 also supports the observed view that students are doing better with the new approach.

The fact that “Global ICT” and the Joint Assessment System have been adopted by all members of staff means that there is now a consistency that would have been difficult to quantify previously. It also shows that the department’s results are fast improving. The fact the assessment system has not been used consistently is an issue, however, this is not unexpected when introducing a new system and as long as this addressed, does not out weigh the advantages.

The effects are more mixed when viewed from an organisational perspective. This has been a valuable “test bed” experience for the school and the fact that the approach has led to increased whole school use of the intranet and has informed choices when adopting a virtual learning environment can only be seen as positive things. However, introducing systems piecemeal could cause some conflict further down the line, if decisions are made either to introduce this on a whole school basis, or introduce a whole school assessment policy that the assessment system will not fulfil.

Even though the system has made it easier for students with poor literacy to access the curriculum, it has made it very clear that in order to make sure that every student is able to do that support has got to be very carefully planned and applied.

However, overall I feel that the introduction of “Global ICT” and the “Joint Assessment System” has been very successful. We have achieved largely what we set out to at introduction and they have resulted in improved performance from learners and the department alike.

With future ICT access, improvements maybe possible to make this approach (or one similar) whole school, however, to be truly successful it must be seen as starting point for a whole school approach to teaching, learning and assessment. The movement towards an asynchronous environment, where students are able to develop the literacies that they will need for the internet age, is a very challenging one. However, in order to have a platform to introduce students to the world of blogs, wikis, podcasts and the writing skills and thinking skills that these will require, this approach will enable this in some way. This will also be the biggest challenge to educators as the skills that we are teaching to the digital natives will need to learnt and understood by the digital immigrants.



[1] School SEF September 2006
[2] School SEF September 2006
[3] Prensky, 2001: Digital Natives Digital Immigrants
[4]Etienne Wenger: Communities of Practice (1998)