Entropy (information theory)
![2 shannons of entropy: Information entropy is the log-base-2 of the number of possible outcomes; with two coins there are four outcomes, and the entropy is two bits.](/Images/godic/202502/20/Entropy_flip_2_coins4109.jpg")
![Entropy Η(X) (i.e. the expected surprisal) of a coin flip, measured in shannons, graphed versus the fairness of the coin Pr(X = 1), where X = 1 represents a result of heads.Note that the maximum of the graph depends on the distribution. Here, the entropy is at most 1 shannon, and to communicate the outcome of a fair coin flip (2 possible values) will require an average of at most 1 bit. The result of a fair die (6 possible values) would require on average log26 bits.](/Images/godic/202502/20/Binary_entropy_plot.svg4109.png")
In information theory, systems are modeled by a transmitter, channel, and receiver. The transmitter produces messages that are sent through the channel. The channel modifies the message in some way. The receiver attempts to infer which message was sent. In this context, entropy (more specifically, Shannon entropy) is the expected value (average) of the information contained in each message. 'Messages' can be modeled by any flow of information.