This is a sequel to my MOOC learning journey post, in which I compared the different online learning platforms and recommended good AI/Math/PM courses. In this post, I want to discuss how using metacognition (to be aware of what/how one’s doing and thinking) can improve learning effectiveness, in terms of absorption and knowledge application. Such self-awareness helps you adjust your strategy/approach as needed. It is like having an alter-ego that zooms above you, instrumenting each process and guiding your next step with rational data.

What makes a good learner?

As shown by the conductor Scott Yoo in this PBS special on music prodigies, there are three building blocks for a good learner,

  1. talent to play music
  2. talent to learn
    • the ability to self reflect
    • the ability to take immediate feedback to improve, they employ a very short feedback loop, compared even with software engineering agile cycle
    • know how to self correct without guidance
  3. motivation to learn
    • know why you like something and how you want to proceed
    • know how to structure your learning

We might not have the innate talent to do something, but we can all learn to learn, i.e., imitate the genius’ learning ability as in the 2nd and 3rd points above.

Be mindful of your own learning style

If you don’t understand what the lecturer says, keep looking for other offerings. It might have nothing to do with your natural disposition or learning ability, but just a mismatch of your learning style with the teacher. For example, I am BOTH a visual and hands-on learner. I cannot listen to hours of proving a formula. I have to first see how it is applied to daily operations, then examine it through visual aids such as animations or charts. The latter is greatly elevated in this era with more visual media available. For example, when I learnt the transformer architecture, Andrew Ng in Transformer Network intuition explained the q, k, v matrics and multihead attention in the encoder in the best manner (for me to understand), because he lays out flatly not the what, but why we even need them in calculation. The 3D Bert poster below takes care of my visual need in understand the how in each time step.

3D illustration of BERT architecture

Learning by video is often challenging for me as I prefer to iterate between reading and doing. Viewing something in a big chunk and practicing in a long assignment isn’t what is most efficient for me. But modern media all focus on video and I have to admit that animation helps tremendously in understanding multi-dimension concepts such as linear algebra and deep learning.

Adopt retrieval practices to improve memory recall

Pulling in the latest research in learning also helps to solidify your newly minted knowledge. Here we’re attempting to overcome the forgetting curve, which illustrates how memory retention declines over time. It was first introduced by German psychologist Hermann Ebbinghaus in the late 19th century. He explained that

  • Rapid Decline: Memory retention drops significantly shortly after learning. Ebbinghaus found that most forgetting happens within the first hour.
  • Exponential Decay: The rate of forgetting slows down over time. After a few days, the decline in memory retention becomes less steep.
  • Impact of Review: Regular review and practice can help counteract the forgetting curve. Techniques like spaced repetition are effective in retaining information longer.

forgetting curve

Spaced repetition

Spaced repetition is a learning technique that involves increasing intervals of time between subsequent review of previously learned material to exploit the psychological spacing effect. This method is particularly effective for improving long-term retention of information. Here’s how it works:

  1. Initial Learning: You start by learning the material thoroughly.
  2. First Review: Review the material after a short period, such as a day.
  3. Subsequent Reviews: Gradually increase the intervals between reviews (e.g., after a week, then two weeks, then a month).
  4. Enhanced Retention: By spacing out reviews, you reinforce the memory just as it starts to fade, which strengthens the neural connections.

For my study, when I start a daily session, I’ll try to write down one thing I remember from the same course yesterday, last week, and last month. Since I am using Obsidian to manage my learning notes, the Spaced Repetition plugin helps to “slip” spacing into my routine easily. The key is to try recalling across different lessons. Per Carpenter and Agarwal’s 2020 research, cramming (intense brain dump the night before exam) only helps short-term memory, but no effect in long-term retention. Spaced practice helps recall significantly as shown in the following image.

recall bar chart

Repeated spaced practice improves recall. Data from from Karpicke & Bauernschmidt (2011). Image Credit: Maverick Learning and Educational Applied Research Nexus

Interleaved practice

Another strategy, interleaved practice which mix different subjects in a session also increase retrieval by 30%, per a study by Rohrer, Dedrick & Agarwal in 2017.

Visual representation of interleaved and blocked practice. The first row of colorful blocks displays different colored blocks or topics of student in randomized order. The second set of blocks displays all blue blocks, then all green blocks, then all red blocks.

Blocking vs. interleaving strategy from Weinstein, Madan & Sumeracki (2018). Image Credit: Maverick Learning and Educational Applied Research Nexus

In my experience, just following the quiz and assignment does not mean that the knowledge has sunk in. To do so, I have to combine the newly gained knowledge with my existing knowledge system, to index, process, mesh, link (just like in Obsidian tool where you can backlink and create a graph diagram of your information collection). Doing the assignment just means “doing and completing”. At the end of the day, it is just 1 task completed.

I also combine PM and AI courses in the same learning session, alternating between different courses to improve connection between concepts, and reduce burnout.

Use output to consolidate learning

In terms of input/output, online courses actually force you to output by having scheduled formative assessments such as in-video quiz (quickly checking if you really understand what is actively being discussed), weekly quiz (to recap the whole module and checking more in-depth reasoning) as well as programming assignment (hands on practice of what you learnt).

We can enhance our retention by mixing different output channels, such as blogging, collaboration with other learners on an open-sourced project, or teaching the same material to another learner (the Feynman method).

From my own programming experience, I learn by trying to solve an existing problem that is beyond my knowledge, I google, read blogs, find some pointers, read official doc to understand how to use some library modules, implement small POC to test on toy data, then gradually read more blogs to understand how to integrate different features. I seldom program the whole algorithm in one shot (like in deeplearning.ai programming assignments). Rather, it is more like piece by piece learning. I also notice that my memory recall for some NLP concepts (e.g., n-gram, perplexity, predicting next word probability, etc.) is way stronger from this class. It is because I implemented the code all by myself without following any prompt, guardrail or lengthy description. This shows that to really sink in knowledge, you have to venture outside the classroom and freely build something on your own.

Improve long term absorption

As my Coursera journey spanned from April to September in 2024, I find it ripe for a checkup now (late Jan 2025). What I did was going through all the courses I took and tried to extract knowledge from them. Here’s what I found:

  • anything that requires no deliberate thinking, design and tangible output (that lasts outside the class) is useless afterwards. For example, quizs are useless as they serve as just instant memory recall right after the course, with no solidification of information into knowledge. Unless you quiz yourself by spacing (into the future), everything is down the drain after the quiz.
  • as long as something casts no effect on your daily life, or make you change in some way, it will fade away. This is in fact how the brain works, by actively pruning unused neural connections.
  • Those with capstone projects in presentations and hands on design work are best in memory retention and practical usage. Specifically, Digital Product Management and AI Product Management both require you to iterate through a complete product planning, design (prototype and code), experimenting and testing cycle. Everything is done by hand and key phases are added in each module. The final delivery is a video presentation which utilizes mixed-modal formats of writing, speaking and teaching, further reinforcing the knowledge. These are the things that can be used immediately at work and yield the best ROI.
  • coding is the most effective, as you can practice and test something new immediately, and whether it works or not is instantly verifiable. With deliberation, you can apply a new tool or paradigm into your current project right after class.

how to really make something your own knowledge?

To combat this, I think it is important to distinguish knowledge domains that require strict memorization, such as learning foreign languages or playing a musical instrument, and those abstract concepts such as economics theories or chemistry that might not easily be applicable to daily life. The former is like scaffolding, things that can be constructed piece by piece by using flashcards and spacing. The latter is harder to integrate into our current knowledge system even after recall, so they stay only in the knowing level that is meaningless. Thus they require totally different strategies.

connecting them to real life experience

For example, if you learn a term, can you do further research and blog about it? Or use it in a presentation of your own project? Or teach someone? It is useless to quiz yourself “what is an indifference curve?” and recite an explanation from memory. It is more productive to find a real world example (ideally somewhere around you) and explain to someone how an indifference curve is driving the phenomenon you see. You will likely be prompted with more questions which will challenge your knowledge base and hopefully expose gaps in understanding. By scrutinizing yourself in this way, your newly gained knowledge isn’t just a superficial definition but something that is deeper ingrained in your life, something that you see as an underlying building block for your daily experience.

For blogging, if it’s just like jotting or “book review” of major ideas, it might be like doing a brain dump that will easily be forgotten. It’s just a bit better than quiz as you have to both recall and write, but no better than asking LLMs to summarize a book! In fact, this was my experience with Mind Maps to organize what I gleam from books before. I thought that organizing my notes into Mind Maps isn’t just neat, it also helps me to restructure what I learn and grasp important concepts. However, what actually happened was that those are still isolated information bits that belongs to the author, and has nothing to do with me. Only after I switched to writing diligently about why is it important, how to apply it for each book, am I able to truly integrate the newly acquired knowledge to become my own.

Even better, if we are able to blog around a theme, it connects the new information to your existing knowledge system. You will be able to visualize the relationship between different concepts and improve your understanding. A theme also needs deeper exploration and explanation (e.g., compare and contrast several concepts, where did they originate, how are they interlinked, etc), not just brain dump of factoids (or run-of-the-mill reporting). What matters is the bridge from new to an existing neural path.

That’s why I deeply admire Yoshizawa Hisako who ran a book club in Tokyo for over 4 decades. Each gathering is around a specific theme and members report on their research. For example, some members even travel to Germany to investigate modern city water usage to report on the theme “water conservation policy”. This definitely cannot be a brain dump from a book, but requires careful study of various materials to come up with your hypothesis that needs real world testing. It is both mixed-modal and immersive learning. The reporting (aka teaching) and q&a only solidifies their learning.

iterating on a project

I also find that maintaining a consistent output cycle by incorporating your newly gained knowledge into your own project and presenting it to another person, adapting and relaunching is the most effective way. With this, we are no longer bound to our classroom but become active practitioners, honing our skills with actual outcomes.

conclusion

I am a bit surprised by my checkup above, but it just confirms what learning experts and psychologists know all along. Finding relevancy and deliberate practicing are key to learning effectively. As I continue my learning journey, I expect to integrate it with public projects that can provide me with feedback and inspiration.