OLI: Closing panel Q&A

March 12, 2008

Andy: fascinating talks covering from current systems improvement to bringing in informal new educational systems. So where should we concentrate? More on changing the current v entering new spaces.

Diana: we do have to do both – there must be a switch from the time spent on formal to a more productive.

John: need to support people who go passed the formal support and refurbishing of skills. OER is probably appealing to older users and to younger and perhaps those in the middle missing out. Should look at the two ends of the spectrum.

Jim: Changing institutions – develop resources to enable productive change. Work tends to be uninformed by how we came to be in the current state. 

Toru: not about abandoning old systems and jumping to new – but do need to increase the overlap from formal teachers to informal learning communities. But they do have the knowledge that is needed to actually support learning.


Judy: Maybe we are being too discouraged –there are a lot of changes. What are the things that are happening that would really help.

Toru: Michael ? at ? presented at Educause about device use not for learning. Great time for learning not for teaching!


Ken: Helping kids become better learners – which 21C learning skills need to be nurtured? Where should learning scientist focus?

John: Social network theory – looking at study groups. Pick up most of what we know from each other. We are guilty of leaving assets lying around – how to capture and index these for use e.g. by alumni. Need to rethink productivity and leverage natural learning mechanisms.


Q: Several communities K-12 v HE. How can we come up with coherent answers.

Diana: Do not think the distinction in sectors applies to her personal work. Ideas tend not to changes but customisation to context is needed. Also need to look at coherence of experience. Make public and exposed may not be enough, citing the OU – exposed material to public but did not align until named degrees.


John: Developing country – need to assist the formal system in developing world rather than the informal. Need to focus on the formal qualification system and align with teaching approach. E.g. civil service exam.

Toru: Agreed that this is an important step.

John: But in India students are coming out of Uni needing to be trained further before work and actually portfolios may be valued more in the creative industries.  He pointed to the moves in Singapore – teach less learn more (actually top down – “you will be creative”)


Marlene: How to get to the explanatory design approach. Already e.g.s of kind of open learning but do not know how they are doing a good job. Need to bring teachers in. Will be a research intensive process to spot what works in social world. Need to tie research to design principles?

Jim: Yes need to do this. Match with learning design approach and work on alignment.

Diana: Science v engineering – but at the moment carrying out design research, producing boundary objects between research and practice. Iterative loops within the research that engage the teachers in the process.

John: Will need to change the way we fund this – turn institutions into Learning Institutions.


Tammy: Creating new assets seems to fit our way of operating rather than reusing. How to get scale.

Diana: Need to see how to best spend time productively. How to find time to look at all that is available if we leave so many assets behind. [Like this one!]. E-learning strategy – most important was how to engage leaders. Not a research issue but a political issue, though tools may expose the issues especially around time and methods.

Toru: Actually this is Tammy’s problem if she is developing smart assessment system – then how to make sure it is used and valuable!

The morning panel is looking at the OER research agenda.


Jim Greeno

Jim talked about the way technology, practice and science can be linked. He referenced Donald Stokes (1995) “Pasteur’s quadrant” as seeing these aspects not as a pipeline but as different dimensions. However even with this recognition doesn’t make it easy to combine these. Looking at

       Learning Science: goals to identify and document phenomena – ending up with “mechanistic” explanations.

       OER: goals to design, develop and distribute resources that support learning.

These appear to be orthogonal axes. The implication to achieve this is that different personnel are needed for these two tasks as relatively independent activities plus the research related work will extend in time. (This has a direct relationship with the organisation of OpenLearn – however with the additional recognition of the whole project as research and everyone in it as “action researchers”.)

Should adopt Explanatory Design Principles as a joint effort – aim for cumulative collection of learning design principles. To get this to happen all write papers, go to meetings etc. The addition then of explanatory design principles as a specific output.

Jim’s next point was on the need to design support for socially interactive learning. He mentioned Activity Theory as helping give a view of the joint learning opportunities over individual methods. This would encourage such things as authentic practice and agency leading to a shift of focus towards design for participation.


Diana Laurillard

Diana’s talk looks at tools for learning design. Seeing this as a route to transfer teachers’ design to student experience. Developing in an echo of John Seely Brown – The social life of teaching. She used the current work on the Pedagogic Planner project to present the analysis tool for breaking down teacher time. She made very nice links to previous talks (Nancy Nersessian on why to use different methods, Martha Lovett on the principles to apply, Marlene Scardamlia on working in design mode, Gerry Stahl on collective activities, Alan Schoenfeld on traces and monitoring, OpenLearn as resource source). Further tools from the project allow analysis of learning outcomes to ensure the topic coverage works, output as information or timetables – i.e. actually being useful to the tutor. Diana also showed how the eventual activities can be represented in LAMS and further generalised to transfer practice.

The manifesto that she ended with is that technology


Toru Iiyoshi

Toru from the Carnegie Foundation talked about the needs of the programme for OERs to bring those outside of education into it. The book “Opening up education” that he edited with Vijay Kumar had gathered together reflections from experts and also given him a chance to think abut the big questions:

       How to encourage participation by learners and educators

       How to extend to niche areas

       What support is needed

Recommendations 1

       New light on hard problems

       Fresh look at education


       New pathways into education

Recommendations 2: culture and practice

       New resources and relationships (e.g. peer tutors)

       Good models for building receptivity

       How to allocate resources

Recommendations 3: sustainability


       Synthesis and synergy

       Governance – widely distributed and collective

Recommendations 4 : make practice and knowledgevisible and sharable

       Community inquiry and discourse

       Build intellectual and technical capacity

Recommendations 5 : Collective culture

Education concierge – guidance and wisdom

What conditions are needed for collective culture

How to create vast network


These come from the executive summary of the book that was also distributed. With aims

       Increase quality of tools

       Effective use

       Advance collective knowledge


I risk therefore I am – quote from somewhere.


John Seely Brown

John offered a point of view on open learning. He pointed out that there are many perspectives and he would look at the institutional level and try to spot the win-win. He started with the long-tail (Pareto principle) for education. 80:20 principle for revenue matched with 20:80 on cost. His example was Boccaccio’s Decameron and the way that the Decameron Web at Brown University had drawn together what is a niche topic. He highlighted also that public access such as at MIT has done helped show the teachers that there was a need to improve alignment between courses once they are out in the open. Perhaps the only way to get academics to look at each others course material.

The same long tail applies to learning – and a “compactness hypothesis” that for any subject there is already a web available group looking the subject because of individual passion in the subject. Perhaps leading to productive inquiry amd spirals of flow. “Learning about” v “Learning to be”. John revisited the amateur involvement in astronmomy as an example – because that actually is his own interest. Astronomy is recognised as a field where amateur and professional cross-over. (I know this  was also picked as a particularly interesting dynamic by Chris Anderson in his Long Tail book.)


He mentioned a new initiative – the worldwide telescope that brings together feeds from many telescopes across the world. A Google Earth for space. He described using this as generating awe! Anyone can generate commented replayable playlists of trips through space and share those – so the learner can be the “curator” of a trip throught he galaxy.


Home Sapien to Home Faber (making things again). With the making done virtually, digitally and digitally augmented physical. (I wonder myself if the new “fibbers” will help this cross-over to the physical world). John also looked at revisiting Dewey’s Pragmatic Technology. A switch from generating “authentic activity” to genuine activity.

He also went back to his Openlearn theme of tinkering going through a “cognitively impenetrable” period to now a rebirth as remix and mashups amd Polyani’s book – attending from v attending to.


The brewing perfect storm of the Hewlett report brings together transformative initiatives into an open participatory learning initiative. He also cited the potential power of making inferences from the data – can we apply the Google analysis techniques to the problems of learning. Enough data could be gathered across the OER community to do this.


As at OpenLearn2007 JSB gave a great talk full of new ideas! He cited a new article that would be wellworth visiting Minds on Fire – Open Education, The Long Tail and Learning 2.0  Educause Review Jan/Feb 08.

OLI: Assessment panel

March 12, 2008

Alan Schoenfeld

This is the session that I am speaking in but first up is Alan – who is a genuine expert on assessment for University of California, Berkeley. Here he is talking about the assessing the right stuff and starts by looking at the discipline aspects at how it can be linked to sense-making. He suggests that the learning community has different characteristics building on knowledge base, problem solving, metacognition., and beliefs etc. His first example is using the magic square as a way to draw learners into solving generalised problems. He briefly considered a community e.g. Brown & ? and the Knowledge Forum work that comes from Marlen Scardamalia’s work. What You Test Is What You Get. (Implies What You Don’t Test Is What You Don’t Get.) His presentation included considering What kind of traces could help with assessment/monitoring? This led inot a challenge for the technology providers to support this.


Mark Wilson

Mark describe how students become dissatisfied with assessments and the general mismatch with exams and expectation. Mark then looked at the results of work on Knowing What Students Know – he showed a triangle with nodes observations, interpretatin, cognition. As the assessment triangle to encourage thinking through of needs looking for coherence.


Assessment design principles:

Model of student learning

Tested items

Clear inferences about student competence

Applied in a particular context


Mark led into looking at the BEAR assessment system that seeks to apply the principles of sound assessment:

       Developmental perspective

       Match between instruction and assessment

       Management by teachers

       Quality evidence



My talk (Patrick McAndrew)

For my talk the slides are available (alongside all the slides for the conference) and there is a video of the whole session as well. They are from the Symposium site at 

Nancy Nersessian

Nancy from Georgia Institute of Technology described a study of university Research Labs to look at ways to encourage “Agentive Learning”. The labs in biological sciences have a need to develop some custom simulations. Nancy had a breakdown into cognitive practice, investigational practice, and interactional practice. Classroom of the future was presented as small rooms with all walls as whiteboards – expect people to bring the technology rather than provide it. Small groups (about 8) are then given complex problems to work on. She gave various examples and came to an end points that to bring out the Agentive Learning Environment:

       learning driven by solving complex problems

       learning is relational

       largely non-hierarchical           

       building (construction of devices, simulations) gives entry

       multiple support gives resilience

Question at the end was how to turn “tinkering” into learning. The answer was as a community of learning – an important point that had not come across in the talk itself.


Gerry Stahl

Started by contrasting the work of cognitive scientists with a focus on the individual with group or community based. His label for this work is group cognition. “We come to knowledge by taking part in collective activities…” Valsiner & van der Veer. His focus was that digitising resources is solved (sort of) but the community aspect of universities has still to be tackled and social networking offers a chance to do something about this. The example he gave was Virtual Math(s) Teams for small group work on problems. The technology level did not look to be highly sophisticated – chat rooms, wiki, document storage. Hiver he suggested a couple of features were important: pointing to things (from chat to whiteboard and from message to message), persistence, and showing current activity (e.g. typing).  The result also displays referencing from the chat to the witeboard. Small group support is then mainly chat and larger sharing and community through the wiki. Suggested challenges are:

How to increase student discourse

How to introduce inquiry learning

How to encourage social networking for deeper kinds of learning

How to integrate – pedagogy, technology, etc.

Why develop a new platform? Needed extra features such as referencing – tried AOL chat as the starting point.


Marlen Scardamalia

Marlene’s position was talking about moving to knowledge creating organisations and took a view of “Belief Mode” to “Design Mode”. With belief mode drives formal education while design mode rarely happens though it drives activity outside formal education. A key difference is the form of questions that people need to ask themselves: a change from is something true, to does it do something useful and advance ideas. Marelen linked to Cathy’s talk about the “New Basics” of education and the move from “Guided Discovery” to deeper understanding. She showed an environment for placing ideas and then linking them together – the environment has a three d model – trying to extract on “Rise above note”. This was linked to Popper’s ideas about ideas but in a way that children can understand. Discourse is brought out as the way to discuss things – with threaded discussion cited as not the way to bring out key points. Instead there ought to be more use of the affordances of technology. Argumentation also may not be the most suitable way to break out of a belief mode. She included some interesting analysis of group dynamics. On the assessment side she also suggested that semantic analysis could spot whether students were using terminology and approaches from experts.

Simon asked at the end about how this work could link with his in looking at coherence. It sounded like that the current work could help do something like this at the person or group level – work needed on the concepts.

Simon had put a fair bit of work into preparing a map of all the ideas that had come up at the Firehose session. The result is


Simon’s talk gave an interesting mix of looking at what the tools (cohere and Compendium) can do for you and how to give yesterday’s ideas some persistence. As I am going to talk about Assessment later in the day it was useful to see how that got a few mentions that I had missed collating myself when I was in the session yesterday. 

OLI: Firehose

March 10, 2008

Simon Buckingham Shum is leading this session and live note taking into Compendium. People each come up and have 2 minutes to say things – grand challenges and ideas. Tammy Sumner for example said what is the learning science equivalent of agile development that has transformed Computer Science? While Neil Butcher of OER Africa spoke about the broader challenges of meeting Africa’s challenges – and to make sure that OERs solve a real problem. He will report back tomorrow and I am sure he will post the Compendium maps as well.

OLI: Physics panel

March 10, 2008

Richard Sohmer

This afternoon is panel based to look at how a Learning Science approach has been used in Physics teaching. The first presentation looked at switching to an “Investigators Club”

Move from “Talk, Text and Identity” to “Talk, Task and Tools”. The talk was described as Accountable talk with 6 ways to get people talking – not to be treated as rules. Tasks are based on the demos to support them based on home available equipment. Tools are to enable children in particular to visualise physics problems. Simulations are used to allow an approach of practice, practice, practice.


Kurt VanLehn

Kurt from University of Pittsburgh talked about design procedure:

Assessment -> tasks -> knowledge -> pathways -> instruction.

Demonstrated by looking at some Physics problems from the Force Concept Inventory comparing qualitative examples with quantitative. The approach is to look for “confluences” when (say) increasing one value causes another to decrease. This is then turned into step based tutoring – which gave gains even when very simply implemented.


Paul Steif

Further Physics examples were provided by Paul with examples of real objects and simulations. He was mainly looking at levers and had a great illustration a truck falling in the river. General approach of evoking familiar experiences can challenge students to give physical interpretations.


OLI: Cathy Casserly

March 10, 2008

I made some shorter notes for this talk as we are involved in the area – but Cathy always manages to introduce new examples and latest data and is great at making connections as she knows what everyone is up to. (And my battery is running out so this post is just before the talk finishes!)

Cathy presents the corresponding primer for OERs to match the information about Learning Sciences already presented. OERs are inspired by open source and a belief that education is part of the common good. Investment in human capital creates a positive multiplier. OER came out of UNESCO workshop in 2002 and has become the dominant label for those working in the field.

Cathy referred to the study that was funded at CMU – 8 weeks online with better results than 15 weeks conventional.

The example used for OPLE was about data gathering in the field. The project seemed rather like the TEL-PI at the OU. Cathy also mentioned the data that Steve Godwin is analysing to pick out the interests of people actually using OpenLearn – the interest in the content, assessment and interactivity. New skill sets include the need to recognise “try, fail and try again”. As an example of the changes in school looking at Hellerup School with a YouTube. Interestingly the video talked about modelling the school as a future workplace – which is very relevant to the work we are doing on our new IET building. The contrast is with current teaching as teaching in “boxes”.

Cathy then connected with the Learning Sciences research to develop deep research, design principles, assessment innovation to link with the OER collaboration, diversity, and ways to tap into the wisdom of the masses.

Herb Simon: One finds limits by pushing them. 

OLI: Marsha Lovett

March 10, 2008

marsha lovett at OLI

Marsha was hit by technical gremlins – no powerpoint slides  for most of the talk but she managed very well! Her talk moves us on to Learning Science. The methodologies that she highlights are to develop comparative studies with random allocation and lab based instrumented studies. However these are difficult to make authentic. So she also talked about what she called “design experiments” where attempts are made within classrooms to improve learning experience. (Perhaps we would call this Action Research?) Difficult though to then impose controls. So the suggestion is to combine methods and OERs offer an opportunity to apply this approach as it is possible to gather fine grained data about experience by automated instrumentation. OERs also give a base of real content for use in experiments. (I very much agree with this as in my previous life I had great difficulty getting data for work on mobile learning and OERs could be just the examples that we need.)

Aspects to measure are

  • Pre and post to measure learning gain
  • Retention measures – how long to be able to retain i
  • Transfer to daily life: e.g. probabilty checks by asking questions about sports
  • Speed of learning on second exposure to the same learning materials
  • Preparation for future learning

People often collect satisfaction or enjoyment. This is indirectly related and needs to be looked at as a factor in time to apply to the learning material.


Marsha then moved on to principles to apply illustrated by statutor. Three principles were picked out:

1. The chance to practice. She used latency in selecting the right answer – but the graph is very bumpy unless more care is taken to extract the separate skills that are being asked. The feedback is then that the experts who combine graphing as a single topic should perhaps be changed. This was cited as an example of how learning science analysis can feedback into the science of learning.

Open questions include:

  • How to represent the skills
  • How do skills change qualitativele
  • How to spot when skills are not being practiced

What incentives are there to complete practice (without “gaming”)


2. New knowledge acquired through lens of prior knowledge.

Experts may well have a “blind spot” in seeing things as a student. This was illustrated with various ways in which experts see problems – e.g for physics novices will match blocks on inclined planes, experts will look for involved equilibrium of energy. She pointed to Richard Mayer’s books on what is need for simulations to work for learners.


  1. Learners refine their knowledge through timely feedback.

Online gives good opportunites for this feedback and link to particular points. Open questions though about how to do this – and in fact if delayed feedback can actually be more effective.


Overall summary – that online student data is a goldmine and be used to inform design, science, and student performance. 

Candace Thille gave an opening about the to the OLI conference – emphasising that it is about the Interplay between the Opening of Learning interpretation of learning sciences with the Open Education world. The focus is on how to bring these together. The aim for the morning seems to include genuine learning rather than just talk.


The first talk is by Mark Kamlet – who confessed to not being a learning scientist but a mathematician. As Provost he had developed links with Hewlett Foundation.

Why do Cognitive Tutors work?

  1. Learning is often sequential and cumulative
  2. Doing is part of learning
  3. Everyone is different

Actually these give insight into ineffectiveness of many approaches – especially the lecture – no chance to pause, very little doing, no chance to personalise.

At this point he mentioned ACTR – not sure what that is.

These ideas though need to be built and tested. He said that investment in the algebra tutor of around $20m. On the other hand the cost is about $500,000 each for Cognitive Tutor lites from OLI. These courses can then pass the achievements of those who built it. Questions remain about cultural transfer – being tried in Arabic and Spanish countries, maybe ok just to translate but actually switch to oral rather than written approach may be key. This needs careful research and evaluation. The usage data needs to feedback into development.


While keen on Cognitive tutor it should not be the only approach.


Just to be online and free is not enough – we need to know how it works


Baumol’s disease: why does cost of education keep going it. Will it go on increasing forever. Baumol’s disease is an economists statement: the cost of something depends on the cost of producing. Productivity keeps increasing in most fields – but not in education! We more or less educate the same way we did 6-700 years ago. So if we match this to most other areas is starts to cost more e.g. wheat production if 1hour of farmer time= 1 bushel 20 years ago and now =3 bushels then the cost of education has to increase by 3 to match if it stands still. (Well now I am Gluten Free who cares about wheat!)


Cognitive tutoring offer a way to reach more people in same time. The arguments for this seem to match to any distance education but I guess the bias to Cognitive tutoring is related to whether there is proven success.


Sustainability: the just make it available may well not work. Not for profit v for profit: actually this is not the same as free v not free (e.g. broadcast TV is “free” higher education is not). Carnegie Learning has been set up on a for profit basis – the main reason for this seems to be the need to reach into schools. Needing sales forces and marketing. There does not seem to be Google-style 3rd stream funding – however at HE level there is a need for Algebra support for e.g. Physics teaching. Offering this into colleges based on retention of students may been it is worth them offering it as free for use. Actually Carnegie Learning is part of CMU so profits would feed in to this. Also he is in favour of open access – citing a move to open access to all academic articles after 12 months. For open education though there are challenges of hitting barriers and the need to have revenue streams for support.