66: HCD Ideation: Prototyping Phase

1-sources

Class 4 Readings in “Design Kit_The Course for Human-Centered Design.” Dropbox. Web. 14 Mar. 2016.

2-what

 

Screen Shot 2016-05-09 at 9.34.57 PM

 

Key Terms:
  • Ideas:
    • answers to the How Might We … ? questions
    • have potential for success, innovative
  • Concepts:
    • polished, concrete ideas
  • Prototypes:
    • tangible ideas
    • rough, just accurate enough to get useful feedback
  • Experience Maps:
    • Visual storyboard that describes beginning, middle and end of user experience of prototype
PROTOTYPING PHASES
  1. Generate Ideas:
    • Choose appropriate space – large enough to move in & post many ideas
    • Gather tools for displaying ideas
    • Recruit large diverse team
    • Keep pacing high energy – no more than 1 hr total, 15-20 min per question
    • Select facilitator
    • Introduce RULES:
      • Defer judgement
      • Encourage wild ideas
      • Build on other ideas
      • Stay on topic
      • One conversation at a time
      • Be visual.  Sketch ideas.
      • Go for quantity
    • Equip everyone to participate
    • Attend to each question, one at a time
  2. Select Promising Ideas:
    • Cluster similar ideas
    • Everyone votes for Top 2.  Vote directly on top of ideas
    • Tally and discuss results.
    • Decide which ideas to develop.
  3. Determine what to Prototype:
    • Break down beginning, middle and end of user experience
    • Create Experience Maps
    • Identify key questions
    • Create order of operations
      • Prioritize questions and prototypes used to investigate these
  4. Make Prototypes:
    • Create rough 3D models of concepts
    • Use digital tools to build a mock-up
    • Role play user experience
    • Create diagram that maps out key processes and services
    • Create stories – newspaper articles, job descriptions that relate key features of concepts
    • Create fake advertisements that highlight key features, tweak to different audiences
  5. Test & Get Feedback:
    • Consider setting – informal or close to actual setting?
    • Define what to test
    • Define feedback activities – ensure that these are best suited to things being tested
    • Invite honesty & openness – communicate that idea is rough and can change in response to feedback
    • Stay neutral – do not be defensive
    • Adapt on the fly – change as you go if possible
    • Provide multiple prototypes if possible
    • Find space and time to discuss initial impressions of feedback
    • Captures ideas & iterations – photo-journal key processes & prototypes
    • Share impressions – compare notes with team and document findings
  6. Integrate feedback & Iterate:
    • Share findings on Post-Its
    • Cluster similar findings
    • Categorize findings by: concerns? pros? suggestions?
    • Evaluation relationship of findings to original intent of product
    • Prioritize feedback – which is most important? what to respond to?
    • Evolve prototypes – makes changes that eliminate barriers and respond to key suggestions
  7. Repeat Steps above Iteratively

 

3-sowhat
This phase enables teachers and students to convert ideas into tangible solutions and then test pieces of these solutions in quick tests that allow designers to quickly learn and iterate from their mistakes.  Going through this process can teach learners how to use iterations to learn and revise and how to learn from their mistakes.

 

4-nowwhat
Preparation Steps
  • HCD Ideation: Prototyping Steps applied to Solving Student Learning Design Challenge
    • Recruit a design team and select a problem that relates to student learning and complete the Inspiration phase
    • Recruit a design team that will help you identify document key insights that you gathered during the Ideation: Synthesis phase
    • Recruit a diverse larger brainstorming team.
  • Scaffolding HCD Ideation: Prototyping Steps for Students
    • Experience HCD process prior to facilitating it to learn how to better scaffold it
    • Let students in design teams select a worthy design challenge (or assign one)
    • Guide them through a student friendly, time affective version of the Inspiration phase
    • Guide them through a student friendly, time affective version of the Ideation: synthesis phase
    • Develop visuals and assign readings that teach students how to go through key steps in a student friendly, time effective version of the Ideation: Prototyping phase.
    • Secure A LOT of Post-Its, secure materials for making 3D prototypes, or teach them how to use Post-It and CAD software.
Early Implementation Steps
  • HCD Ideation: Prototyping Steps applied to solve student learning design challenge
    • Meet with brainstorming team in a space with lots of Post Its and wall space.
    • Generate ideas while following the brainstorming rules.  See above.
    • Select the most promising ideas. See above.
    • Investigate the user experience and decide what to prototype.  See above.
    • Create prototypes of various forms.  See above.
    • Test prototypes and get feedback.  See above.
    • Integrate feedback into revised prototypes and iterate See above.
    • Develop, test, and refine prototypes several times.  See above.
  • Scaffolding HCD Ideation: Prototyping Steps for Students
    • Facilitate student design teams through a student friendly, time-effective version of the steps listed above
Advanced Implementation Steps
  • HCD Ideation: Prototyping Steps applied to solving a student learning design challenge
    • After enough has been learned from prototyping, move on the Implementation phase.
  • Scaffolding HCD Ideation: Prototypin Steps for Students
    • Let students reflect on how Inspiration: Prototyping Steps can be used to develop better insights to problems in other courses and in their own lives
    • Facilitate students through the Implementation phase

 

5-relatedstuff

65: HCD Ideation: Synthesis Phase

1-sources

Class 3 Readings in “Design Kit_The Course for Human-Centered Design.” Dropbox. Web. 14 Mar. 2016.

 
2-what
Screen Shot 2016-05-09 at 9.39.46 PM
Key Terms:
  • Learnings:
    • highlights from interviews, observations, and anecdotal notes from Inspiration Phase
    • communicated in full sentences to capture whole stories
  • Themes:
    • categories that organize learnings
    • capture similarities in multiple data sources
    • headlines for clusters of similar learnings
  • Insights:
    • succinct expressions of what was learned
    • offer both new and expected perspectives
  • How Might We …?:
    • generative questions based on insights that trigger brainstorming
  • Ideas:
    • brainstormed items that answer How Might We … ? questions
    • during brainstorming these ideas are not judged
    • can be represented by quick sketches
 
HCD Ideation Phase: Synthesis Steps
  1. Capture Your Learnings:
    • Set up a download space: space with lots of room for displaying Post-Its
    • Download learnings: 
      • document key findings on Post-Its
      • print out key photos
      • what to capture?
        • personal details – who? age? location? profession?
        • interesting stories
        • motivations
        • barriers, frustrations
        • interesting interactions with people and the environment
        • lingering questions
    • Share inspiring stories:
      • tell compelling stories
      • be descriptive and specific
      • share the who, what, when, where, why, how
      • identify resonant stores
      • Tips:
        • actively listen – compare / contrast stories, identify recurring themes
        • capture information in small pieces – quotes, succinct stories
        • display notes – display and organize Post-Its on wall by data source details
  2. Search for meaning: identify themes that can inform design
    • Don’t feel shy about retelling key stories
    • Cluster related information:
      • each picks top 3
      • cluster Post-Its in categories such as: consistent problems, shared significance, surprise elements
    • Find themes
      • name clusters
      • rearrange until all themes have been identified
    • Turn themes into insight statements
      • each theme can have multiple insights
      • express as complete sentences
    • Revisit your challenge
      • revisit initial design challenge using insights
      • narrow insights using design challenge to top 3 – 5
  3. Create “How Might We … ?” Questions (generative questions)
    • multiple questions per insight
    • use plain simple language
    • aim for proper scoping
      • too narrow – hinders creativity
      • too broad – not actionable
    • suggests starting points
    • suggests key logistics
    • select Top 3
      • trust gut
      • select most generative questions, not necessarily the easiest ones

 

3-sowhat
The Human Centered Design (HCD) Process is an approach that can be used to develop projects that engage students AND teach content & 21st Century skills. The Ideation: Synthesis phase converts research into meaningful actionable insights.  It converts learnings into design opportunities.  These are skills that students and teachers can develop in order to learn how to better interpret and leverage data.

 

4-nowwhat
Preparation Steps
  • HCD Inspiration Steps applied to Solving Student Learning Design Challenge
    • Select a design problem that relates to student learning and complete the Inspiration phase
    • Recruit a design team that will help you document and identify key insights that you gathered during the Ideation: Synthesis phase
  • Scaffolding HCD Inspiration Steps for Students
    • Experience HCD process prior to facilitating it to learn how to better scaffold it
    • Let students in design teams select a worthy design challenge (or assign one)
    • Guide them through a student friendly, time effective version of the  Inspiration phase
    • Develop visuals and assign readings that teach students key steps in student-friendly version of the Ideation: Prototyping phase
    • Secure A LOT of Post-Its or teach them how to use a post-it app
Early Implementation Steps
  • HCD Ideation: Synthesis Steps applied to solve student learning design challenge
    • Meet with design team in a space with lots of Post Its and wall space.
    • Record these learnings on Post-Its.  See above for details.
    • Work with data team to identify recurring themes. See above for details.
    • Work with data team to identify key insights. See above for details.
    • Work with data team to formulate and prioritize “How Might We … “ questions. See above for details.
  • Scaffolding HCD Inspiration Steps for Students
    • Facilitate student design teams through a student friendly version of the steps listed above
Advanced Implementation Steps
  • HCD Ideation: Synthesis Steps applied to solving a student learning design challenge
    • Gather more research on lingering questions
    • Repeat early implementation steps again
    • Then move unto to Prototyping phase
  • Scaffolding HCD Inspiration Steps for Students
    • Let students reflect on how HCD_ Inspiration Steps can be used to develop better insights to problems in other courses and in their own lives
    • Facilitate students through Prototyping phase
5-relatedstuff

57: From Framework to Protocol

1-sources
This link will only work if you’re logged into Echo.  Sorry, non-New Tech readers.  If you have more questions, you can tweet Jim May at @jimamay.

 

2-what
Why have a focus area for change:
  • we get better SPECIFICALLY, not GENERALLY
Finding a Focus Criteria:
  • related to STUDENT LEARNING
  • emerges from multiple sources of data
  • politically feasible
  • advances school’s mission & purpose
 
The Framework can be used to classify types of focus areas:
framework

 

Caveats:
  • If focus is not shared enough – don’t know where to look to see if it’s working
  • Hope is not a strategy
  • Principal can not be the only source of feedback (need to leverage peer feedback)
Guiding Questions
  • What are the unique challenges that relate to focus area? Evidence of these?
  • What are the unique strengths we already possess relating to focus area? Evidence of these?
  • What do possible strategies and outcomes look like?
  • How will we observe and measure outcomes related to focus area?
From focus to strategy:  Sources:
  • experts in the field,
  • reading literature,
  • disciplined experimentation of new strategies
 
Strategies (Goor characteristics):
  • Can have multiple activities that share one focus area
  • Addresses both technical and adaptive elements
  • Increases learning of stakeholders related to focus area
Driver diagrams: 
  • displays primary drivers (red) and related factors (blue) that connect to focus area (green);
  • can be used to brainstorm strategies (lavender) that relate to focus areas
driver

 

Testing and Learning from Strategies:
  • Use student work as primary source of evidence
  • Analyze student work using specific question(s) related to focus area
  • Examine the learning task that led to the student work
  • Keep time between analysis and action very short
  • Try to link data directly to improving practice
  • Use data analysis to develop a deeper shared understanding of focus area
 
Designing & Interpreting Data Inquiries:
  • data related to specific questions tied to focus area
  • shared knowledge base that can be used to make analytical decisions about data
  • create artifacts that anchor sense making
  • impose structure on data to make meaning of it

 

3-sowhat
School change is hard, but vital.   Developing and implementing processes for setting, implementing, learning from, and revising school change goals is vital to maintaining and improving the quality of PBL schools.  Having a specific focus related to student learning that is backed by data derived from student work helps schools select focus areas that serve their primary clients, the students, and their primary mission, learning.

 

4-nowwhat
Preparation Steps
  • Brainstorm possible focus areas using criteria above.
  • If needed, gather more data related to potential focus areas.
  • Analyze school structures that relate to focus area (drivers).
  • Research & develop potential strategies for improving and learning more about focus areas.
  • Identify evidence that can be used to assess potential strategies.
  • Develop a data inventory that describes all the data sources that will be used to investigate focus area, when these will be gathered, how to access these, and ideas for how to interpret these.
Early Implementation Steps
  • Implement strategies.
  • Use pre-planned evidence gathering strategy, to collect student work that can be interpreted to learn more about focus area.
  • Create data summaries that make sense of data and provoke interesting conversations.
  • Have regular data meetings focused on making sense and developing key questions from data related to focus area.
Advanced Implementation Steps
  • Involve students and alumni in key committees that select focus areas and that interpret and learn from data.
  • Use student panels to gain more qualitative data that relates to other data sets collected to investigate focus area.
5-relatedstuff

55: PBL Trade-Offs

1-sources

 

2-what

 

Screen Shot 2016-05-09 at 10.27.04 PM

 

Big ideas/transfer VERSUS specific knowledge and skills:
  • Err on the side of big ideas -> knowledge may be too abstract to be useful
  • Err on the side of specific -> may learn skills but not why / when to use them
  • Aim for a balance between discrete information and meaningful conceptual frameworks
Complex realistic messy performances VERSUS efficient and sound tests:
  • Err on side of messiness -> may lead to confusion, take too much time
  • Err on side of efficiency -> tests lack validity, learning may end up being inert and useless in real contexts
  • Mitigate some of the messiness of performances with models and rubrics
  • Use tests for evaluating discrete enabling skills
Teacher control VERSUS learner control of the work
  • Err on side of teacher control -> students lack voice and choice in their work, may lose engagement
  • Err on side of student control -> students lack knowledge to design frameworks that tie to key learning goals
  • Let teacher control substantive holistic curriculum choices
  • Involve students in choices about product formats and day-to-day (during day) flow
Direct teach VERSUS constructivist approaches
  • Err on side of constructivism -> may take too long, may skip details in discrete skills
  • Err on side of direct teach -> students don’t make meaningful connections among concepts and skills, can not grasp big ideas
  • Use more direct approaches for discrete straightforward knowledge
  • Use more constructivist approaches for deep subtle concepts
Depth VERSUS breadth of knowledge
  • Err on side of depth -> may not get to enough ideas in course
  • Err on side of breadth -> knowledge learned at level that is too superficial for transfer
  • Develop a yearlong plan that prioritizes important ideas
Uniform VERSUS personalized work and expectations
  • Err on side of uniformity -> lack of differentiation may mean that not all students get opportunities to learn material
  • Err on side of personalization -> unsustainable, not all student learn goals
  • Use rubric for final product and allow choices that do not change the rubric at all (or very much)
  • Cluster student needs and plan remediations and enrichment activities around these
  • Use varied methods to teach the same content
Effective VERSUS merely engaging
  • Err on side of effective -> student learn skills without sense of ownership
  • Err on side of engagement -> students may have fun without learning anything
  • Design activities that are both effective and engaging
  • Make sure all hands on activities are aligned to learning goals
  • Make sure drier lessons are explicitly connected to more engaging work
Complex VERSUS simplistic
  • Err on side of complexity -> leads to confusion and lack of transfer
  • Err on side of simplicity -> big ideas lose meaning, accuracy & transfer
  • Aim to craft essential (Driving) questions that are both age appropriate and meaningful
Well crafted plan VERSUS flexibility & openness
  • Err on side of rigid planning -> may miss out on good learning opportunities, may not make necessary adjustments to students’ needs
  • Err on side of openness -> plan may lack clarity & focus, learning may feel inefficient, disconnected and untied to a larger purpose
  • Develop a well crafted plan and use formative feedback to refine it
Great individual unit VERSUS larger goals 
  • Err on side of single unit -> may neglect other important school goals
  • Err on side of larger goals -> may develop a unit that is too diluted to be effective
  • Design units that are aligned to school and curriculum goals
  • Use textbooks and tools in the service of well designed learning plans

 

3-sowhat
Project design is hard.  There are many balances to strike while designing and implementing effective and engaging projects.  Any PBL educator that’s designed and implemented several projects has probably faced at least 1 or more of the trade-offs above.  Diagnosing which trade-offs keep plaguing one’s design can give hints as to what types of design solutions teachers need to research in order to create more balance in their projects.

 

4-nowwhat
Preparation Steps
  • If a project went well, try to notice how one managed the trade-offs above in order to extract general tips.  Apply these tips to continue to design successful projects.
  • If a project went poorly, try to identify which of the trade-offs above broke down.  Research and brainstorm how to better manage that trade-off and apply those tips to future project designs.
Early Implementation Steps
  • Be aware of how trade-offs (above) are impacting project implementation and note how you manage these.
  • Survey students using some the trade-offs above to check if the right balance is being struck.
Advanced Implementation Steps
  • Use diagnoses of one’s past successes and failures, to identify which tradeoffs one manages poorly.  Develop checklists associated with those trade-offs that help guide one to a better balance in future projects.
  • Ask students for advice in your troublesome trade-offs on how you might strike a better balance.
5-relatedstuff

53: Coaching Lenses

1-sources

 

2-what

Screen Shot 2016-05-09 at 10.37.09 PM

Coaching Lenses
Inquiry Lens:
  • Values questions, multiple sources of data
  • Problem solving
  • Admits to not knowing everything
  • Aware of assumptions and limitations
  • Concerned with the quality of the question
Change Management Lens:
  • How will change be made?
  • What conditions are needed to create change?
  • Is change possible?
  • Analysis of change conditions
  • Strengths and gaps of current climate
  • Identify and leverage change opportunities
  • What incentives, resources, and skills are needed to promote change?
Systems Thinking Lens:
  • Schools (and classrooms) are interconnected complex systems
  • Systems have logical outcomes
  • Conflict is natural
  • Complexity and diversity are healthy
  • What are the stuck points and energy sources in the system?
Learner Lens:
  • More experienced learners have more starting and sticking points
  • Considers prior knowledge and experiences of learners
  • Sets realistic important objectives that involve direct concrete applications
  • Provides individualized feedback
Systematic Oppression Lens:
  • Prejudice is a notion based on limited information
  • Racism is a product of beliefs and systems that are situated in history, economy, politics and society
  • Who has power (and not)?
  • How does power affect the truth?
  • How does power affect safety?
  • Who’s missing from the leadership?
Emotional Intelligence Lens:
  • Self awareness and self management
  • Social awareness and relationship management
  • Can ask for help and receive feedback
  • Adaptable and flexible
  • Can manage stress
  • Can identify beliefs
  • Welcomes change
  • Reacts well to setbacks
  • Can empathize
  • Can identify social networks and power relationships
  • Can create safe environments
  • Can appear to various learners
  • Good at conflict management
  • Can collaborate well
Recommended read:

 

3-sowhat
This list of lenses was initially intended for instructional coaches, but also applies well to PBL facilitators.  To successfully manage a PBL environment, PBL educators need to play many roles besides teacher.  They need to model and teach 21st century skills.  They need to be effective leaders and good project managers.  They need to learn how to groom student leaders.  The lenses make the many roles of PBL educators more explicit.  Knowing these lenses and the different approaches that go with each can help PBL educators apply the right skills and tools to the right problems.

 

4-nowwhat
Preparation Steps
  • Reflect to identify which lenses one applies often and why
  • Reflect to identify lenses that one applies well (and not) and the impacts of those strengths and gaps
  • If gaps might have negative consequences on teaching, research strategies to mitigate those gaps
  • Identify how one uses strengths in lenses to solve problems.  Brainstorm and research ways to extend those strengths
  • Identify a worthwhile problem that one can solve or learn more about by applying 1 (or more) of the lenses
  • Use different perspectives to develop hypotheses and potential solutions to focus problem
Early Implementation Steps
  • Trial solutions or gather data related to hypotheses in the classroom
  • Reflect on how solutions work (or not) or how gathered information supports (or does not support) conjectures based on looking at the problem from different lenses
Advanced Implementation Steps
  • Recruit thought partners who are strong in lenses that are your gaps
  • Ask how thought partners see your focus problem and what insights they have – develop conjectures and possible solutions related to these
  • Teach lenses to students when the perspectives build into these lenses can help them think in ways that develop their understandings and their products

 

5-relatedstuff

52: What the Hack?

1-sources
Note: This link will only work if you’re logged into Echo.  Sorry, non-New Tech readers.  If you want to contact the presenters, you can try tweeting them at  @leeafleming and @mpacheco11.

 

2-what

 

Screen Shot 2016-05-09 at 10.41.51 PM

 

Challenges and Pitfalls of Change Efforts:
  • solutionitis – inflamed, too many solutions
  • jumping to solution implementation before unpacking it, the quick fix
  • overplanning, underdiagnosis
  • can lead to change fatigue and risk aversion
  • confirmation bias – looking only for the YES in the feedback loop and not the NO’s
Hackers
  • know systems deeply
  • try several things – persistence
  • innovate
  • set up low stake tests to try out new ideas
Hacket Tips:
  1. Chunk it down
  2. Think outside the box
  3. Use small tests
  4. Shift your mindset – learn, not plan
Recommended Reads:

3-sowhatTeachers are constantly problem solving.  They are also role players in broader solutions that affect the whole school.  A hacker’s approach to problem solving could help stakeholders solve problems in innovative and tested ways.

4-nowwhat
Preparation Steps
  • Find a problem connected to student learning that is worth solving
  • Research and brainstorm solutions related to the problem
  • Research and brainstorm ways to gather data that test various solutions
Early Implementation Steps
  • Trial solutions and gather data on their effectiveness from students and other methods
Advanced Implementation Steps
5-relatedstuff

51: Amping Up the Authenticity

1-sources

 

Note: This link will only work if you’re logged into Echo.  Sorry, non-New Tech readers.  If you’d like to get access to more related information, check out Kevin Gant’s blog at https://intrepidedblog.wordpress.com/

 

2-what

 

Screen Shot 2016-05-09 at 10.46.06 PM

 

Evolution of projects from towards more authenticity:  scenario based -> project world = real world

 

Characteristics & Challenges of Most Authentic Projects:
  • Project work = work of the world
  • Student products are used by people outside school
  • Students have more choice to define products
  • Not all standards can appear in all products
  • Can give all students opportunities to gain mastery of all learning targets in project scaffolding
  • Involves a provocative driving question that acts as a call to action for students to solve a real problem
  • Driving question can be resolved  by a wide variety of end products
  • Content authority ceded to experts outside the classroom
  • Time frame can get mushy for some products due to interactions with real clients running on schedules different from the school schedule

 

Related Useful tools:
  • Nepris –
    • Web-based tool that connects experts to classroom, teachers can post requests and repris matches experts to request
    • Heard that wait time is about 3 weeks
  • IGNITE by DiscoverSTEAM
    • Online platform that sets up secure communication platform between companies that want to interact with students and schools
    • Companies present projects to students that they want solved; if students develop solutions the companies can use, they can pay students
    • Teachers can work with companies to develop rubrics aligned to standards
    • All communication between students and companies is recorded and reviewed for quality and research purposes
    • IGNITE is FREE to schools because companies pay for it as a way to recruit American talent and to market their companies
    • Companies prefer to have a yearlong relationship with students – my idea to extend the interaction time with companies is to relay the company project through different courses;  project could start in ELA for initial research and writing processes, then move to a Science course for designing data studies and gathering data, then move to a Math class to analyze the data, then move to a Social Studies class or Audio Visual class to finalize conclusions and create products, etc
    • NOTE: This company is looking for PBL teachers to pilot their tool.  Contact them if you’re interested
3-sowhat

The more time teachers and students spend doing PBL projects, the more difficult it gets to make projects feel fresh and meaningful.   Facilitating real world projects is one way for advanced PBL teachers and advanced PBL students to frame learning within more relevant contexts and challenges.  Authentic problems can also give students exposure to careers they have not yet considered and give them a better idea of how real experts solve problems.

4-nowwhat
Preparation Steps
  • Extend your network – contact companies, university groups, and local groups of experts who do things that relate to your course
  • Build a database that contains potential contacts – see above
  • Contact them early – in the summer if possible – to see if there are problems they’d like students to solve or explore that could align to standards
  • Check if school has partnered with companies like Nepris or DiscoverSTEAM – if the partnership is there, learn how to use those tools and start using them expand your network
  • Plan Year at a Glance (Scope and Sequence)
  • Contact experts early to see if they can interact with students at the appropriate times in the year (or adjust your scopes & sequence as needed)
  • Develop templates (email and phone call) for students to contact experts
  • Develop tools that students can use to document their unique projects and their ties to key content and problem solving processes that are being uncovered in the classroom
  • Design scaffolding that supports real world project work and aligns to standards
  • Research and commit to a design process (or similar problem solving processes) that will organize the project and that students will use to solve project problems
Early Implementation Steps
  • Work with experts to support students with content knowledge and with formative feedback they can use to improve their understandings and products
  • Organize project around a common design process (or similar problem solving processes) that students will apply to develop different products of their choice
  • Implement class-wide scaffolding that ensures that all students have opportunities to learn targeted content
  • Be flexible with final product completion and final presentation dates as these may vary depending on their solutions and their clients
  • If possible – set a common proposal presentation date to review and give critical friends feedback on proposed solutions
Advanced Implementation Steps
  • Develop sustaining relationships with experts that enable different cohorts of students to gain exposure to real world problems that have better and better project design (if the opportunity arises to interact with same experts on similar projects form year to year)
  • Arrange for field trips that allow students to get close hand  exposure to experts and to gain experiences that can help them gather useful data related to their projects
  • For experts that like prolonged relationships with students, collaborate with other courses to extend project over time (see relay idea above)
5-relatedstuff

49: Building Assessment Literacy

1-sources

 

2-what

 

Screen Shot 2016-05-09 at 10.52.39 PM

 

Principles for Interpreting Assessment Results:
  • Sampling Principle of Testing – tests are not direct measures of mastery; conclusions about mastery in a domain are based on inferences drawn from a smaller sample
  • Discrimination – assessment items that discriminate, separate students who know from students who don’t; use these items to reveal differences that actually exist
  • Measurement error – many sources – examples: differences in questions, different student moods, etc.
  • Reliability – conditions that give consistent results
  • Score inflations – scores inflated by cheating, luck, etc ..
Accounting for Sampling & Measurement Errors:
  • Measures and attach error bands to test scores
  • Keep test scores in perspective – tests have limitations, they don’t assess all important skills, course grades can not be compared across schools like standardized tests
Different Ways of Reporting Performance:
  • Beware of raw scores – not normalized for difference of questions between tests
  • Norm referenced tests – PRO: results are normalized by results of all testers, CON – improvements are easy near bottom and harder near the top
  • Criterion referenced tests – measure mastery of specific skills, PRO – gives specific information CON – vulnerable to sampling errors due to small number of questions per skill
  • Standards-Referenced tests – tests based on content / performance standards.  PRO – gives specific information relating to standards CON – vulnerable to sampling errors due to small number of questions per skill
Trade-Off
  • Reliability vs detail – more detail tends to smaller sample sizes which tend to be less reliable
    • it’s risky to draw conclusions from small sample sizes – e.g. standards assigned to only 1 – 2 questions on a test
How to Measure Improvement
  • Cohort-to-cohort – compares same grade, different students – CON – susceptible to demographic biases
  • Value-added, longitudinal – compares same students are different times – CON – susceptible to a lot of error PRO – works well with cumulative subjects
Strategies for Interpreting Data
  1. Look beyond 1 year with cohort-to-cohort or value-added data
  2. Compare results to relevant district and state data
  3. Compare results to most recent assessments on similar standards
3-sowhat
Knowing sources of data error can teach one to use data selection, summarizing and interpretation strategies that minimize impacts of data error.  Teachers planning to investigate data with students need to know sources of data error in order to teach them to students and to teach students how to make careful interpretations of data.  Using the strategies for data above can help highlight important data features by using comparisons to relevant reference groups.

 

4-nowwhat
Preparation Steps
  • Decide what data sources will be summarized.
  • Select methods for summarizing data that compare data to relevant reference groups – these can include comparisons to other schools, other districts, state scores and to the same students at another time in the school year
  • Script questions stakeholders (students) will use to investigate the data.  For more details, see this article.
  • Create data displays that provoke data discussion.  See this article for more details.
Early Implementation Steps
  • Facilitate data meeting with students that has them use reflection prompts to draw conclusions about data displays as individuals, in pairs, and as a whole group
  • Brainstorm with students how to further explore key questions and next steps for improving key trends
Advanced Implementation Steps
  • Gather feedback from students aimed at improving data conversations and use these to fine tune future discussions
  • Have students brainstorm data goals that can be measured with specific upcoming assessments
  • Have students brainstorm methods for gathering more reliable data related to group academic goals
5-relatedstuff

48: Building Data Organizations

1-sourcesChapter 1 in Boudett, Kathryn Parker., Elizabeth A. City, and Richard J. Murnane. Data Wise: A Step-by-step Guide to Using Assessment Results to Improve Teaching and Learning. Cambridge, MA: Harvard Education, 2005. Print.

2-what

 

Screen Shot 2016-05-09 at 10.56.50 PM

 

This book was intended for school administrators.  I think it can also apply to teachers.  Here I summarize parts that are generalizable to all settings so that both teachers and administrators can benefit from the book.
Two Question to Ask:
  1. Are you satisfied with the way you capture info garnered from assessments?
  2. Are you satisfied with how you gather information from multiple data sources for your students?
Building Data Organization Steps:
  1. Build a data inventory: a chart or spreadsheet that organizes and lists all data sources that will be interpreted to improve student learning.  Columns on data inventory sheet could include:
    • data source name (hyperlinked)
    • content area
    • collection date(s)
    • students assessed
    • accessibility – how to get to the data
    • current data use
    • possible more effective use
  2. Build a Data Team(s)
    • identify well-connected stake holders who would be interested in investigating data to improve student learning
    • types of data teams:
      • organizing data – small team
      • interpreting data – big, collaborative effort – the more who are involved the larger number of stakeholders have ownership in school improvement
  3. Create a Schedule that Allows for Regular Collaborative Work
  4. Plan for Productive Meetings: 
    • establish norms – example – no blame, approach data as a learner
    • use protocols to structure conversations
    • adopt an improvement process: small groups analyze data charts and discuss what they notice, share key findings on chart paper, use structured protocol for formulating questions about why data looks like that, whole group – establish consensus on most important questions to address
    • plan agendas for meetings
3-sowhat
Organizing data teams to summarize data and organizing data meetings to interpret data and formula key questions from the data is a process that can improve student learning at the staff level and at the teacher student level.

 

Compiling a data inventory and writing out current and possible uses for data is an exercise that can make teachers and administrators more aware of how they are currently leveraging data and what they can possibly do to make more use of the data.

Teachers generate A LOT of data when they comply with school-wide grading expectations. Teachers can make more use of this data by creating compelling, summarizing data displays and using these displays to facilitate small and whole group discussions with students on data inferences, possible emergent questions, and possible followup experiments to further investigate or improve the data.  This process can make students feel like more valued stakeholders in the education process and could help students set and achieve individual and community level academic goals.

 

4-nowwhat
Preparation Steps
  • Create a data inventory of past and upcoming data sources that are worth interpreting
  • Research and/or brainstorm protocols for discussing data
  • Create data displays that are compelling.  These could
    • NOT reveal confidential individual student data
    • show how learning of student group(s) evolve over time
    • mastery levels of student group(s) tied to key big ideas
  • Develop an agenda that has time for students to interpret data as individuals, in pairs, and as a whole group
Early Implementation Steps
  • Facilitate a project reflection (or semester reflection) that features the data discussion agenda developed above
  • Record key findings of students
  • Work with students to achieve consensus on key questions and methods to investigate these and possible next steps and methods to verify that these are working
Advanced Implementation Steps
  • Form a data organization group that includes students after running several data meetings.  This committee can advise teacher on how to summarize data that makes it more easy to identify compelling questions and trends
  • Survey students to check about whether or not data meetings are making them feel like more valued contributors to the learning community
5-relatedstuff

45: PBL Dilemmas & Resolutions

1-sources

 

2-what

 

Screen Shot 2016-05-09 at 11.10.24 PM

 

The dilemmas facing educators trying out the backwards design approach to unit planning are shared by educators trying out Project Based Learning (PBL).  The following lists some possible resolutions that can be used to confront naysayers (both outside and inside our own brains.)

 

Misconception: We have to teach to the test
Possible Resolutions:
  • Design authentic projects that are aligned to test standards
  • Use test blueprints to streamline curriculum so there is time to go for depth in key standards

 

Misconception: We have too much content to cover
Possible Resolutions:
  • Use projects to have students solve complex problems that make the connections between ideas and skills explicit and vital
  • Remember that teaching is not the same thing as learning
  • Use standards to streamline (not extend) the curriculum so there is time for depth
  • Use textbooks as references, not as the syllabus

 

Misconception: This work is hard and I don’t have the time.

 

3-sowhat
Standardized tests are limited measures of understanding.  Training for the tests is like training for a doctor’s physical instead of aiming for better health.  Research has shown that authentic assessments and pedagogy better prepare students for high stakes assessments than drill and kill methods.

 

The textbook is not the curriculum.  The TIMS test has shown that curricula that aim for depth over breadth teach more math concepts and skills.

 

It’s OK to start small.  Cook one gourmet per year or per semester and do it in collaboration with other teacher stakeholders.  Build with the help of other educators and refine using feedback from students.

 

4-nowwhat
Preparation Steps
  • Analyze and prioritize standards
  • Identify which standards clusters lend themselves well to authentic tasks and pedagogy
  • Design a project aground standards using backwards design template and standards.  If possible, collaborate with other teachers on this design.
  • If possible, review design with a student panel before launching project.
 
Early Implementation Steps
  • Launch and facilitate entire project.
  • Take notes on what’s working and not working during the project.  Use these tips to refine project.  Generalize this feedback in order to apply them to future projects.
  • If full blown projects are too intimidating, try designing and implementing inquiry based lessons.
Advanced Implementation Steps
  • Collaborate with other educators to build and refine a database of projects.
  • Design a PBL course that uncovers most standards using projects
5-relatedstuff