Ket vectors are a type of vector used within quantum physics to define the vector space (which is technically called a Hilbert space), which is used to mathematically describe the set of states within quantum theory. These ket vectors are also sometimes just called kets. A ket vector A is written using the following notation in quantum physics:
|A>
These ket vectors represent column vectors within the vector space, so they're somewhat different from more conventional vector mathematics.
The mathematical principles in operation while working with these vector spaces are usually taught broadly under the subject heading of "linear algebra," which is why physics students typically must take a couple of linear algebra courses in order to earn even an undergraduate degree in physics. Another complicating factor is that the terms within these vectors are complex numbers, which definitely makes things more ... complex.
Why the Weird Name?
These are called ket vectors because they are closely linked with another type of vector, called the bra vector, which is denoted as:
<B|
If you were to take the inner product of the bra vector B and the ket vector A it would be denoted as:
<B|A>
Therefore, this notation is called bra-ket notation, which is a play on words for "bracket notation," because the inner product is contained within brackets. This, apparently, is the sort of thing that makes mathematical physicists chuckle.