Top videos

Why is KL Divergence not symmetric?

Generative AI
8 Views · 7 months ago

The KL divergence of distributions P and Q is a measure of how similar P and Q are.
However, the KL Divergence of P and Q is not the same as the KL Divergence of Q and P.
Why?
Learn the intuition behind this in this friendly video.

More about the KL Divergence formula:
https://www.youtube.com/watch?v=sjgZxuCm_8Q

Shannon Entropy and Information Gain

Generative AI
8 Views · 7 months ago

CORRECTION: at 13:41, the probability is 6.1e-5 and not 4.8e-4 (however, the entropy is 1.75, which is correct). Thank you @dlyChimi!

Learn Shannon entropy and information gain by playing a game consisting in picking colored balls from buckets.

Announcement: New Book by Luis Serrano! Grokking Machine Learning. bit.ly/grokkingML
40% discount code: serranoyt

Accompanying blog post: https://medium.com/p/5810d35d54b4/

0:00 Shannon Entropy and Information Gain
2:22 What ball will we pick?
4:33 Quiz
5:06 Question
5:14 Game
7:17 Probability of Winning
7:45 Products
11:00 What if there are more classes?
12:34 Sequence 2
13:44 Sequence 3
14:57 Naive Approach
15:34 Sequence 1
19:44 General Formula

Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction

Generative AI
8 Views · 7 months ago

Correction: At 30:42 I write "X = Y". They're not equal, what I meant to say is "X and Y are identically distributed".

The variance is a measure of how spread out a distribution is. In order to estimate the variance, one takes a sample of n points from the distribution, and calculate the average square deviation from the mean.
However, this doesn't give a good estimate of the variance of the distribution. The best estimate, however, is obtained when dividing by n-1 instead of n.
WHY!?!?!?!?!?!?!?
In this video, we dig deeper into why the variance calculation should be divided by n-1 instead of by n. For this, we use an alternate definition of the variance, which doesn't use the mean in its calculation.

*[0:00] Introduction and Bessel's Correction*
- Introducing Bessel's Correction and why we divide by $ n-1 $ instead of $ n $ to estimate variance.

*[0:12] Introduction to Variance Calculation*
- Explaining the premise of calculating variance and introducing the concept of estimating variance using a sample instead of the entire population.

*[1:01] Definition of Variance*
- Defining variance as a measure of how much values deviate from the mean and outlining the basic steps of variance calculation.

*[1:52] Introduction to Bessel's Correction*
- Discussing why we divide by $ n-1 $ when calculating variance and introducing Bessel's Correction.

*[2:35] Challenges of Bessel's Correction*
- Sharing personal challenges in understanding the rationale behind Bessel's Correction and discussing my research process on the topic.

*[3:20] Alternative Definition of Variance*
- Presenting an alternative definition of variance to aid in understanding Bessel's Correction and expressing curiosity about its presence in the literature.

*[4:45] Quick Recap of Mean and Variance*
- Briefly revisiting the concepts of mean and variance, demonstrating how they are calculated with examples, and explaining how variance reflects different distributions.

*[7:05] Sample Mean and Variance Estimation*
- Explaining the challenges of estimating the mean and variance of a distribution using a sample and discussing why sample variance is not a good estimate.

*[8:49] Bessel's Correction and Why $ n-1 $ is Used*
- Explaining how Bessel's Correction provides a better estimate of variance and why we divide by $ n-1 $ instead of $ n $. Emphasizing the importance of making a correct variance estimate.

*[10:51] Why Better Estimation Matters?*
- Discussing why the original estimate is poor and why making a better estimate is crucial. Explaining the significance of sample mean as a good estimate.

*[13:02] Issues with Variance Estimation*
- Illustrating the problems with variance estimation and demonstrating with examples why using the correct mean is essential for accurate estimates. Explaining the accuracy of estimates made using $ n-1 $.

*[15:04] Introduction to Correcting the Estimate*
- Discussing the underestimated variance and the need for correction in estimation.

*[15:57] Adjusting the Variance Formula*
- Explaining the adjustment in the variance formula by changing the denominator from $ n $ to $ n - 1 $.

*[16:22] Calculation Illustration*
- Demonstrating the calculation process of variance with the adjusted formula using examples.

*[16:57] Better Estimate with Bessel's Correction*
- Discussing how the corrected estimate provides a more accurate variance estimation.

*[18:24] New Method for Variance Calculation*
- Introducing a new method for calculating variance without explicitly calculating the mean.

*[20:06] Understanding the Relation between Variance and Variance*
- Explaining the relationship between variance and variance, and how they are related mathematically.

*[21:52] Demonstrating a Bad Calculation*
- Illustrating a flawed method for calculating variance and explaining the need for correction.

*[23:37] The Role of Bessel's Correction*
- Explaining why removing unnecessary zeros in variance calculation leads to better estimates, equivalent to Bessel's Correction.

*[25:08] Summary of Estimation Methods*
- Summarizing the difference between the flawed and corrected estimation methods for variance.

*[26:02] Importance of Bessel's Correction*
- Emphasizing the significance of Bessel's Correction for accurate variance estimation, especially with smaller sample sizes.

*[30:19] Mathematical Proof of Variance Relationship*
- Providing two proofs of the relationship between variance and variance, highlighting their equivalence.

*[35:24] Acknowledgments and Conclusion*

Thanks @mkan543 for the summary!

Small Language Models Under 4GB: What Actually Works?

Generative AI
8 Views · 7 months ago

Never get stuck without AI again. Run three Small Language Models (SLMs)—also called Local LLMs—TinyLlama, Gemma-3 and Phi-4-mini—completely offline; all fit in 4 GB or less and work on any laptop and older hardware.

────────────────────
🔧 Hardware & Software used
• Laptop Ryzen 5 4500U, 8GB RAM, Ollama (no GPU needed!)
• Phone iPhone 13 Pro with Mobile PocketPal AI (local GGUF)

────────────────────
🔗 Model resources
• ChatGPT global outage (news)
https://timesofindia.indiatime....s.com/etimes/trendin
• Phi-4-mini reasoning paper
https://www.microsoft.com/en-u....s/research/wp-conten
• TinyLlama 1.1 https://huggingface.co/TinyLlama/TinyLlama_v1.1
└ GGUF Q4_0 637 MB https://huggingface.co/TheBlok....e/TinyLlama-1.1B-Cha
• Gemma-3 https://huggingface.co/blog/gemma3
└ GGUF Q4_K_M 0.8 GB https://huggingface.co/Maziyar....Panahi/gemma-3-1b-it
• Phi-4-mini https://huggingface.co/microso....ft/Phi-4-mini-reason
└ GGUF Q4_K_M 2.5 GB https://huggingface.co/lmstudi....o-community/Phi-4-mi

────────────────────
🎬 More on local AI
• End of VRAM? https://youtu.be/M9ZphDPRP_w
• Is local AI image generation dying? https://youtu.be/ad7jBaNgIW8

🛠 Support the channel
Patreon https://www.patreon.com/NextTechAndAi

────────────────────
▼ Comment Poll
What would YOU use offline AI for?

#SmallLanguageModels #LocalLLM #OfflineLLM #LocalAI