vectors that are "perpendicular", or "orthogonal", have, in some sense "nothing to do with each other".  There is no way you can approximate or "get closer to" [0,1] using a multiple of [1,0]...at all, really!  Similarly, vectors that are linearly independent, in some sense, can each differentiate themselves from all the rest.  They might approximate each other a bit - they can be somewhat similar - but all of them contribute something unique to the mix that no combination of the others have.  For example, you can approximate [1,3] using a multiple of [0,1] (or vice versa), but you'll always have a little something missing.

 

When working with vectors that have a lot of dimensions, you can get a feel that this sense of vector "similarity" becomes a lot like the statistical concept of "correlation".  For example, [1,2,3,4...] is a lot like [2,3,4,5..], both when interpreted as vectors and when interpreted as data.  But [-2, -1, 0, 1, 2] and [2, 1, 2, -3, 4] have very little to do with each other, both interpreted as vectors (they're orthogonal) and interpreted as data.

 

When thinking about it in terms of vectors, the degree to which two vectors approximate each other has something to do with the angle between them.  When the vectors point the same way, this is "maximum" similarity.  When they point in perpendicular directions, this is "no" similarity.  When pointing in opposite directions, this is sorta like "inverse" similarity.  This sounds a lot like the measure of "similarity" should be the cosine of the angle, which you probably already know is closely tied to the dot product.

 

And in fact, this is exactly the definition that is used for statistical "correlation".  Put one set of data in one vector, the other set of data in another vector.  Write out the formula for the cosine of the angle between the two vectors, (v1 dot v2 / (|v1||v2)), and this is exactly the formula for correlation.

 

 

We can define a similar concept for *functions* that are "orthogonal", or "perpendicular", or "linearly independent".  Functions that cannot approximate each other are in a sense "uncorrelated".  How do we define "correlation" in this sense?  A lot like the original statistical definition of correlation: Consider what happens if you "sampled" the first function n times, and did the same for the second function.  Now treat each sample as "data" and apply the statistics definition of correlation, i.e. using the dot product.  As the number of "samples" goes to infinity, the summations turn into integrals; thus, the correlation between the functions is <integral of (the two functions multiplied together), divided by the integrals of each function>

 

Intuitively, you should think of orthogonal functions as functions that, if you created data from them, they would not seem correlated at all.

 

We can find a series of "orthogonal" functions that all have the same "can't approximate each other" property that a basis of orthogonal vectors has.  This is an "inner product space", which comes from the alternate name for a dot product ("inner product").

 

Here's one:

(hmmm...polynomials?)

 

Actually, just like there are lots of orthonormal bases for a single space of vectors, there are lots of orthnormal bases for the functions.  You might recognize some of them:

 

<constant, sin t, cos t, sin 2t, cos 2t, sin 3t, cos 3t, ...>

<the atomic orbitals: s, p, d, f, ....>