What Is Maximum Entropy In Simple Words?

 

Questions:

1. What is meant by maximum entropy?

2. What is meant by a maximum entropy classifier?

Answer to 1:

Imagine 3 buckets with 10 red and 10 blue balls as shown in the picture. The first and last have well separated (or ordered) blue and red balls, and have low entropy. The middle one has uniformly placed red and blue balls (unordered or random) and has high entropy.

So, maximum entropy is achieved when we have uniform distribution of things or in other words when they have the most randomness.

Answer to 2:

Maximum entropy classifier as the name suggest is related to maximum entropy. It is a classifier which prefers the uniformity or maximum entropy if no data is observed. But as it sees the data, it has to move away from the maximum entropy by explaining data. After it has explained the data, it again tries to maximize the entropy on whatever remaining is not seen.

Alternative Answer:

You can think of maximum entropy as the closest you can get to pure nonsense (random letters all with the same frequency of occurrence). 

It is the densest amount of information you can get across given a fixed set of letters. 

For example if you have only two letters "A" and "B". 

AAAAAA would be low entropy. 

ABAAAAA would be in the middle and 

ABABAB would be the the maximum entropy for the alphabet.

https://www.quora.com/What-is-maximum-entropy-in-the-simplest-terms

Maximum Entropy Example In JavaScript

PLEASE WAIT WHILE THE CODE PROCESSES THE DATA