I would say most B-sides are not from the album, although sometimes they do come from the B side of the LP. Usually they are different versions, unreleased songs, and sometimes live versions. Many times, groups will release B-sides and other rare tracks as an album later. I have quite a few of these, and they often contain some gems.
These are rare tracks because singles have been always relatively harder to get than LPs. A deep cut, however obscure, is not usually rare, because one can always stroll into a music store and buy the Pearl Jam album "Ten" for the song "Jeremy" and still get the deep cut "Garden." But you may have more difficulty finding the song "Yellow Ledbetter," the B-side of the "Jeremy" single. So they released the 2 disc "Lost Dogs" B-sides and rarities compilation, containing that track...