The Expectation Gap Survey

WONKHE and Pearson today released the analysis of their second Student Expectation Gap survey. The survey was available throughout December 2020 and covered English and Welsh universities with 3,389 student responses. Students have understood the situation academics are in and are satisfied with their responsiveness to feedback and support requests; however, only 40% agree that their experience as been of sufficiently good quality.

What we take from the findings is that among the students we surveyed, the fundamentals are generally in place. Teaching staff seem to be (mostly) engaging and responsive, and though some students flagged specific frustrations about learning remotely, most reported good access to learning resources.
WONKHE

The responses showed that 46% of the courses were delivered entirely online, and a further 14% started with some face-to-face and then moved entirely online during the term. Only 33% of student had campus-based sessions throughout the period. 80% of the students have less than 10 hours of timetabled sessions per week, and 17% had less than two hours (mostly PGT), the rest of their couses were independent study.

The pandemic has accelerated the move to technology-enhanced learning. According to this survey, students are open to keeping the changes once the government lifts the social distancing rules. Universities now have the challenge of assessing what delivery looks like post-COVID. They must decide what should be retained in the short term, what to develop for the longer-term strategically, and what to remove.

The survey suggests students want:

More significant interaction between students on campus and supplemented online through discussion forums
More contact time with tutors in the classroom, online in seminars, through remote check-ins with tutors, and via email.
Encouragement and support to become independent learners through online formative self-assessment, more frequent assessments, and progress reviews indicate how they perform on the course.
A more consistent approach to teaching across modules
The campus and classrooms used for interactive tasks and activities, practical experiences, lab-time, and fieldwork.
Online learning used to add flexibility, remove constraints around scheduled contact hours, and enhance learning delivery.
A better User Experience UX design of the VLE to improve signposting and to set expectations around learning.
Content broken into manageable chunks interspersed with a large variety of activities and knowledge checks.
Online access to wellbeing, careers, and academic support services.
More skills development through independent study learning activities for academic writing, digital learning, project and time management, the confidence to engage with groups, information literacy, and independent learning.

You can read the summary and the research findings on the WONKHE website.

Group size and interactions in online courses

The Open University (OU) in England was set up in 1969 by the UK government to widen access to higher education. The university has over 160,000 students, almost all studying ‘off-campus’, currently categorised as distance learning in the HESA data, but this term may need updating. The OU has had a long-standing principle of splitting cohorts into groups of 25 students. With almost all UK courses currently delivered entirely online due to a lockdown, I want to know what effect group size has on interaction levels? Is there an optimum group size for highly interactive online courses?

Cohort numbers are important as we want to run courses with lots of interaction where students engage in active and collaborative learning that improves their outcomes. It is vital to keep costs down by controlling the volume of staff interaction provided, so classes are sustainable and represent value for money. We also want a balance for students with opportunities for interactions, but they do not feel lost and disconnected.

My first search found a great quote from a 1969 paper from The Journal of Social Psychology; ‘...as group size increases, individual participation decreases.‘ While this paper looked at on-campus, free discussion within small groups, it was a good starting point. With groups of two students, they have to be highly engaged, whereas groups of five provide individuals with a space to hide or take a step back.

However, anecdotally from my teaching days, sometimes larger groups can create exciting conversations and develop a social norm of participation that does not happen in smaller groups. I assume that optimum group size might differ for synchronous and asynchronous learning activities, between different pedagogic approaches, teacher expectations and interaction levels, and technical and non-technical subjects.

Group sizes

I found some recommended size ranges include Sieber (2005)‘s 12 for instructors new to teaching online and Tomei (2006)‘s suggestion of 12 for postgraduate courses. Colwell and Jenks (as cited in Burruss, Billing, Brownrigg, Skiba, & Connors, 2009) suggest an upper limit as 20 for undergraduate and 8 to 15 for postgraduate. In a paper by Parks-Stamm et al. (2017), student interaction in classes of 14 or fewer students increased with more instructor participation, but this mattered less with larger groups of 15-30 students. Orellana (2006) states that 16 was perceived as the optimal group size by academics teaching online to achieve the highest level of interaction.

An Inside Higher Ed article interviewed several American universities with established online portfolios asking about optimum group size. The University of Massachusetts at Lowell have 28,000 online enrollments; they cap their undergraduate classes at 27 and postgraduate courses at 25 students. Granite State College in New Hampshire keep group sizes between 12-15 students, and on the other end, Brigham Young University at Idaho’s average class size is 37. The WCET a digital learning policy group for universities sets a ‘rule of thumb’ of 20-25 students.

Initial recommendations

I could not find anyone in my short search that recommended group sizes of over 27 students, but there were many suggestions that group size is not the best metric to use. Starting with the OU’s suggested groups of 25 students and then monitoring each is a good starting point. You can then monitor student performance, withdrawals, instructor response time, engagement measures, including the volume of student/instructor interactions, and student feedback. This data will allow you to assess if the group size, interaction levels, and course design meet the students’ learning and social needs. You could also provide regular opportunities for small-sized groups, including 2-3 students working together for students who would benefit from more intense interactions.

Point A to Point B

The shortest distance between two points is a straight line.
Archimedes

Much of the work we do in educational technology helps people understand where they are, where they want to be, and then support them to achieve it. In the book Intervention, Dan John‘s process working with athletes has many parallels with our work with academics.

Some teams, departments, or universities know precisely where they want to be-Point B, but they are not clear on where they are now, Point A. In this situation, our job is to identify their current position, then create a plan to reach their goal. Others know exactly where they are but need help to see a realistic goal, requiring ideas, standards, and progressions. A third more common group is unrealistic about their Point A and/or Point B and needs help to identify both before making a plan and starting work. We need to know both point A and point B to draw the line between them.

There are things that everyone we work with needs; ideas of innovative practice to improving student experience, more straightforward and better-integrated technology, comprehensive training and support, and a clear development process. We also need effective project management and a schedule that takes into account the academic calendar. But some tools can help assess where a team is on their journey and the next step in their progression, such as the Quality Matters Standards, the OLC’s Quality Scorecard, and the SAMR learning model.

But,

If people know the goal, assess where they are and connect the dots.
If people know where they are now, but either want an unrealistic goal or do not know what they want, show them the next step and connect them.
With everyone, always focus on the process and the keys to success.

If you are currently working on your service offer, spend some time on a set of questions and a collection of principles to find Point A, Point B, and the most direct route. A systematic approach to educational developments will help you find the straight line.

Sixteen hours per week of deliberate practice

I have been reading and watching a lot of the late Charles Poliquin this week. In a video this morning, he talked about the amount of learning you need to do to be world-class at what you do:

Eight hours per week is the minimum you need to learn… The people that make the most money in their profession learn sixteen hours a week… The more you know, the more you realise you don’t know.
Charles Poliquin

Charles Poliquin was well-read and based all his recommendations on expert knowledge, so I spent some time looking for this recommendation’s origin. I returned to the ‘Role of Deliberate Practice in the Acquisition of Expert Performance‘ paper by K. Anders Ericsson to see how these recommendations for developing masters compare. The article presents research from multiple sources that it takes ten years or more of necessary experience to develop the skills to produce outstanding work at a world-class level. This practice should be time-limited at 2-4 hours per day, every day, for many years. This recommendation was present in research on experts from chess, musical composition, mathematics, tennis, swimming, long-distance running, scientists, authors, and poets.

The average age of the first published works was 25.2 for scientists, and 24.2 for authors and poets and the average age at which they published their most remarkable work was 35.4 for scientists and 34.3 for authors and poets. There was an average of more than ten years between the scientist and authors first work and their best without considering the time it took learning and writing to get to the first publication. The highest performance levels were not attained just by years of experience but by deliberate efforts to improve slowly over a very long time.

Deliberate practice is a set of activities that are most effective in improving performance. It requires the motivation to do the task and effort to improve performance. These activities are repeated consistently with slight variation and should provide immediate informative feedback. The idea of deliberate practice in developing scientists and artists is very similar to athletes and musicians’ development. This development involves years of intensive preparation under an expert teacher, total emersion in the field, and most importantly, identifying the most likely activities to result in the desired achievements.

When looking at scientists, the highest performing are also those who produce a larger number of publications than others in their field. This would suggest that writing to develop arguments would be the deliberate practice that helps them develop a new published theory or idea. Writing is a demanding activity; most world-class scientists stick to a rigid daily schedule that involves writing as the first significant activity of each morning and is time blocked to 1-2 hours, leaving the rest of the day to other work.

In virtually all domains, there is evidence that the most important activity – practice, thinking, or writing- requires considerable effort and is scheduled for a fixed period of time during the day. For those exceptional individuals who sustain this regular activity for months and years, its duration limited to 2-4 hours a day, which is a fraction of their time awake.
K. Anders Ericsson

Two to four hours per working day would be equal to ten to twenty hours per week. To hit Charles Poliquin’s sixteen hours of learning per week target, we would require just over three hours of learning or around 40% of a typical eight-hour working day. The question then is, what is a Learning Designer’s deliberate practice that will allow them to become world-class, and how do I provide an environment to help Learning Designers do this deliberate practice to gain mastery?

The length of time between each iteration of a course makes the day to day work of a Learning Designer unsuitable as deliberate practice, so I need to find something more immediate. My wife is launching a company and becoming active on social media to market the brand. The kinds of skills she is learning are very similar to those that make an excellent Learning Designer—developing Learning Designers as Youtubers might be an effective strategy. Youtubers produce regular video content that is published, continually work to improve all aspects of quality, operate on social media and interacting with viewers to drive traffic to their youtube site, and using the analytics tools to track activity and inform future content. This might be a crazy idea, but it might just work.

Using Abbing’s brand model to develop a service offer

University leadership teams are currently planning what delivery will look like next academic year. A form of blended learning will likely be maintained even if social distancing rules are relaxed. Educational technology and academic development teams will need to restructure their services to provide academic departments with the support they need to transition from this year’s delivery model to a more sustainable and quality-driven model for the future. But what does that service offer look like and how can it be designed to provide freedom for academic teams to explore what this new future looks like?

Author/Copyright holder: erik roscam abbing. Copyright terms and licence: CC BY-NC-SA 2.0

Erik Roscam Abbing’s brand model could be used as a starting point for Edtech teams to create their new service blueprint. The starting point is to map out the team’s own identity, vision, mission, and behaviours. An understanding of the Capability Maturity Model can also input into the team’s desired brand. I have added below my current thoughts on the first phase for my team. If you have any questions or want to collaborate on ideas, get in contact with me on Twitter @samueljtanner.

Team Identity

We have moved towards a Learning Design skill set in the team rather than the more traditional Learning Technologist. Each member of the group would consider themselves as a ‘techie’ and has an expertise that sits somewhere in the nexus of three core technical skills; Learning and teaching, multimedia and technology development, and design. Learning Designers operate as project managers, follow design thinking methodologies using personas and prototypes, and adopt a scholarly approach to quality assurance and continuous improvement practices.

Vision

We believe in the transformational nature of technology, and that learning and teaching can be made better when technology is used to design student centred experiences. Teachnology allowed learning and teaching to be:

Flexible: accessible to anyone that wants to learn, at whatever stage of life they are at, and whatever their context.
Personalised: designed to meet students individual goals and provide choice as these change.
Active and collaborative: engaging learning experiences that prepare students with the skills they need for the workplace, including problem-solving, teamwork, communication, and resilience.
Redefined: using technology to create student experiences previously impossible with physical constraints.

Mission

By 2025, all students will have a flexible, personalised, and active and collaborative learning experience that uses technology to provide better learning outcomes.

Behaviour

We are:

partnering with academic teams to co-design modules and courses
defining what quality looks like and how to get there sustainably
sharing ideas of what is possible and what works
building an easy to use and seamlessly integrated technology ecosystem that provides the tools needed

My ideas will be different from yours

The ideas here are just a brain dump around the direction I am taking my team, but I suggest using the same framework for your institution. Phase two will look at the identity, vision, mission, and behaviour of those teaching at university. My team is a service for academic departments to help them teach students, and so our customers are the lecturers. It is a time of disruption for the role of academics, and the answers to the questions in phase two will be very different now than six years ago when I moved from further education to the university sector. I have some research to do, but I imagine that brand promise will be something along the lines of…

Brand promise: Your Learning Designer will help you design, develop, and deliver a flexible module quicker, easier, and provide a better student experience than if you had done it independently.

Design-driven companies perform better

The DMI Design Value Index measured design-focused publicly traded companies’ performance against the larger stock market. For a company to be counted as a design-led, it must meet six design management criteria:

The organisation must be publicly traded in the US for 10+ years.
The scale of the design organisation and deployment is an integrated function.
Growth in design-related investments and influence have increased overtime.
Design is embedded within the organisational structure.
Design leadership is present at senior and divisional levels.
There is a senior-level commitment to design’s use as an innovation resource and integrative force.

These criteria assessed whether design was a long term strategic priority, built into the organisation’s structure, and well resourced through hiring practices, facilities, and technologies. Qualifying companies needed a design-focused C-level executive and CEO and leadership team publicly recognising the importance of design in their work and publicity.

Simply put, design is a method of problem-solving. Whether it is an architectural blueprint, a brochure, the signage system at an airport, a chair, or a better way to streamline production on the factory floor – design helps solve a problem.
The Design Management Institute

The DMI, funded by Microsoft, identified fifteen companies that meet all six criteria, including Apple, IBM, Nike, Procter & Gamble, and Starbucks. Results showed that design-led companies outperformed the S&P 500 index by 228% over ten years.

Motiv and DMI developed eight ways in which a company could improve its performance through design:

The Wow factor
Brand expression
Solving unmet user needs
Develop better customer experiences
Rethinking strategy
Hardware/software/service integration
Market expansion through persona development and user understanding
Cost reduction

If you want to know more, view the DMI’s Design Value Index and it’s Design value system that makes use of the Capabilities Maturity Model.

Email lists and course enrolments

The primary tool for independent knowledge workers building a portfolio job and interacting directly with their customers is the email list. Even if the target is 1000 true fans, the email list will likely need to be much larger. Regular, quality content will attract an audience, and that audience can lead to a living. With this in mind, the first job for an independent creative is to build and cultivate an email list by giving away content that people value.

Many subscribers will be interested in free content but may not want to buy anything or might be highly price-sensitive based on their circumstances. Converting subscribers into purchases requires build relationships through regular interaction, so they value and trust the content that will eventually come with a price tag. Teachable advise those course creators who follow their marketing process have achieved a minimum of 2% conversion rates from their email lists to enrolments on online courses, but this could be much higher for established email lists with cultivated relationships.

Teachables Course launch goals calculator

Revenue goal = (Total email subscribers x Conversion rate) x Price for your course

Teachable recommends that creators aim for a premium price for their course to allow the instructor to spend time creating and running a great course and making it easier to reach any target revenue goal. Teachable’s top ten instructors’ average course price is $272 (£200), but some courses can cost over $1000, where significant interaction with the instructors are a feature.

If our revenue goal is £10,000 from the first run of our new course, and we price it using the average from Teachable’s top ten instructors, we need to sell to 50 students (£10,000/£200). Using the goal’s calculators conservative estimate of 2% conversion rate, we first need to build a list of 2,500 email addresses before we market and launch our course.

The Course launch calculator is based on real courses data, but it is not meant to be an exact formula. Teachable’s aim is similar to Kevin Kelly’s objective with his 1000 true fans idea, to demystify the process of making money on the internet and encouraging people to start creating content that someone somewhere will love.

Regularly create and release content someone will value, collect the email addresses of these people, interact with them, and over time you will be able to make a living working for yourself.

Project Management Basics

The first step towards a mature development process for developing online courses is to introduce some project management basics. According to the Project Management Institute, a project is a temporary endeavour undertaken to create a unique product, service or result. For a set of tasks to be a project, there must be a start and end date and produce a one-off output. The three core tasks for managing a project are cost and effort estimation, task allocation, and risk management.

Resist the ad hoc. Announce that this is a project, and that it matters enough to be treated as one.
Seth Godin

Here are ten suggested actions to get you started with managing your projects:

Set a start and end date and identify your critical path; each task’s last possible completion date allows the project to hit the end date. Identify the dates by work backwards from completion and take into account tasks dependent on the completion of another. Track task completion against this critical path, and do what is needed to hit all the deadlines, so the project ends on time.
Develop a way of estimating the cost and effort involved in your projects. Understand how complexity, size, and reuse impact these estimates. Start with what you can find in literature and then review and update it after each new project is complete to improve its accuracy over time.
Keep a list of risks, possible consequences, and likelihood and introduce ways to reduce the chance of them happening to minimise disturbances during the project. Considered project risks that affect the schedule or resources, product risks affecting the final course or module, and business risks that affect the university.
Assign a Project Leader responsible for the project; this should be someone who controls the critical resources such as the Academics line manager of Head of Department. Regularly communicate with the Project Lead and get sign-off from them on crucial decisions.
Produce a project schedule that includes all the tasks to be completed and their due dates, any key milestones, and gates where the key project team members get sign-off to progress. Add the critical path and start and end dates and get everyone to sign it off before work begins. Update it as things change.
Send out weekly project highlights to the Sponsor/owner and Project Leader. Use a traffic light system to help them identify if they need to intervene. If in amber or red, add a brief note saying why it is in trouble and what is required to bring it back to green.
Make all your work visible and share it will the whole project team. Show the critical path, the estimates, the risks, and the schedule. Keep a record of all the weekly highlight reports and the other documents in a central location that the project team can access. Produce regular prototypes in various forms as soon as possible and regularly afterwards, share it with the intended students for feedback before the course launches.
Write down everything. Record everything that people expect and everything that people promise. Let everyone know you have recorded it. Keep a log of what you’ve done and how. You will need it when things go wrong or when planning the next project.
When working on multiple projects, keep a complete list in one place. Use the weekly highlights traffic light system and record the next action to move each forward. If you manage a team, get the members to do the same and keep a central list of all the projects and their status.
Evaluate your projects when you sign them off. Create a lessons learned document and get the project team to list what worked and what didn’t. Integrate any changes into the process for next time.

CMM and online learning development process design

Universities need to significantly increase their capacity to develop high quality online and blended delivery through the recruitment and training of Learning Designers. Institutional scale requires a shift from focusing on individual Learning Designers’ capabilities to concentrating on the organisation’s capabilities for designing learning. First, universities must consider how Learning Design projects are managed and implement sound project management principles. Next, they need to implement a structured development approach through research, evaluation, and peer review, the creation of rigorous quality standards, a formalised development pipeline, a strong community of practice, and progressive professional development.

Good project management of course design and development projects keeps them delivered on time, on budget, and within scope, and ensure a high standard for the student experience. Most learning development models are in their infancy, with few standards defined. If institutions want to produce novel and innovative online courses, they need to borrow design and development techniques from other fields, including software engineering.

The Capability Maturity Model (CMM), developed at the Carnegie Mellon University for large software projects, evaluates a product development processes level of maturity. It is focused on standardising the process of design and development and so counter to many agile methods but will work well with established teams in large organisations. CMM accepts that design and development processes are idealistic and do not represent most projects’ messy and improvised nature, but that tightly controlled and fully documented processes are better. The messiness level varies from project to project, and CMM aims to categories these into five levels of maturity.

Learning Design teams can use CCM’s five levels to improve their operations and assess how individual Learning Designers perform. Teams work through the levels in sequence to standardise their process to produce consistently high-quality online courses no matter the team working on it. The highest level would be represented by a clearly defined process that can be taught and learned, with clear quality metrics that lead to near-zero adverse outcomes. It includes mechanisms for capturing innovative practice and incrementally improving with each course iteration.

The five stages of maturity

All stages above level two subsume the standards of the previous level.

Level 1: Initial – an ad hoc process which can be chaotic. Each Learning Designer follows their version of a basic process. This is the starting point for using a new or undocumented repeat process.

Level 2: Repeatable – each project includes cost scheduling and basic project management practices. Some processes are repeatable, with some consistent results.

Level 3: Defined – the process for managing and developing courses is standardised and documented.

Level 4: Managed – measurement is made of the process and course quality. These measures are used to control and improve practices. Effective achievement of the process objectives can be evidenced using metrics.

Level 5: Optimising – processes are continually improved through quantitative measures and testing innovative ideas and new technologies. (Few developers are considered to be at this level).

The next three to five years will see massive growth in online learning, and universities core delivery will keep much of the changes they have implemented over the last ten months. Departments responsible for supporting online and blended learning should be spending time now on process improvement to optimise their design and development model to prepare for this rapid growth.

Get in touch with me on Twitter if you want to discuss the process of design and development of online learning.

Generating ideas with brainstorming

Brainstorming is the commonly used method to generate ideas. It is a group activity where the group’s collective thinking is used to come up with many ideas—booking a specific time allocated to brainstorming highlights to attendees that they have a defined period to generate ideas, and that the evaluation will come later. The technique first appeared in Alex Osborn’s 1942 book How to think up.

Ideation: the formation of ideas or concepts.
Oxford Languages

Brainstorming is a problem-solving process used to activate prior knowledge, develop possible theories or hypotheses, and identify things to research further. The session’s aim should be to construct a shared model to explain the problem and provide a clear direction for what to do next. Design Thinking, Design Sprints, and Problem-Based Learning (PBL) all make heavy use of brainstorming as a collaborative idea generation method.

In PBL, once questions have been identified from the trigger material, the group brainstorms what they already know and identify potential solutions. The group then analyses and structures the brainstorming session’s output and uses missing knowledge to create learning objectives. Each group member then independently researches the objectives, and then they come back to discuss findings.

The Interaction Design Foundation rules for brainstorming

The Interaction Design Foundation has eight rules for running brainstorming sessions:

Set a time limit.
Start with a question, a plan or a goal – and stay focused on the topic.
Defer judgement or criticism, including non-verbal.
Encourage weird, wacky and wild ideas.
Aim for quantity.
Build on each others’ ideas.
Be visual.
Allow one conversation at a time.

The key to generating new ideas is a challenging question or problem statement, get that right, and the rest is down to the people you get in the room. Have a single, specific problem statement that expresses a point of view or a question that challenges the groups’ assumptions.

Do not let anyone evaluate any idea or answer as they are created. Coming up with ideas needs to be dynamic so shorter sessions of up to 60 minutes work best and force people to focus on new ideas rather than evaluating them. If you choose to go longer, never have more than 90 minutes without a break.

Encourage as many crazy, wacky, and alternative ideas as possible; it is a quantity session, the quality will be developed from the ideas at a later session. Whiteboards with markers or post-it notes on a wall make it easy for people to follow and inspire more ideas, so prepare the room before you start.

Switching between the two modes of individual and collective ideation sessions can be seamless—and highly productive. Alex Osborn’s 1950s classic Applied Imagination gave advice that is still relevant: Creativity comes from a blend of individual and collective ideation.
Interaction Design Foundation

Brain dumping – the brainstorm for individuals

Brainstorming is a group activity whereas the brain dump is its solo equivalent. By dumping all your ideas about something onto a page and out of your head, you open up mental space to more creative ideas.

You can use individual brain dumps and group brainstorming together for even better results and make sure everyone contributes. Getting each group member to do a brain dump at the start can help quiet group members contribute and free up everyone’s headspace for new ideas. You can use brain dumps after a brainstorming session to continue the conversation after the group session. Allocate five minutes at the start of the session for each member to write questions or ideas on paper or post-it notes and then share their thoughts, put these in a visible place and group together any duplicate ideas.

Try the better brainstorming technique

Hal Gregersen published a three-step method to better brainstorming sessions in the Harvard Business Review. The technique focuses on questions rather than answers to get group members excited about the challenge presented and avoids the negative traps present in many ideation sessions.

Step 1: Set the stage

Select a challenge you care deeply about and invite people that will see that challenge from fresh angles. Set our the problem in a maximum of 2 minutes, this will keep it high level and avoid constraining the questions with too much context.

Set two rules

Questions only
No preambles or justifications to questions

Step 2: Brainstorm the questions

Set a timer for 4 minutes and ask a group to generate as many questions as possible within this time. Aim for 15 questions in the 4 minutes to keep the pressure up and the focus on questions only. Once complete, check with the group about how they feel about the challenge, if needed, rerun the 4 minutes until the group is excited.

Step 3: Identify a quest – and commit to it

Study the questions and select a few interesting ones. Try to expand these questions with a set of follow-up questions related to them. Once the challenge is fully understood, and multiple approaches have emerged, commit to at least one pathway.

Let me know on Twitter if you give this a try. I will be doing a brain dump and then group brainstorm for my project 4W/KG in the coming week and will share the outcome.