Invariant theory lives to describe the behavior of group actions on polynomial rings. The study began trying to describe binary forms. However, extending the techniques used, into the realm of multivariate polynomials, met many obstructions. Enter the dawn of the 20th century and David Hilbert.
A linear algebraic group $G$ is some Zariski closed subgroup of $\operatorname{GL}(V)$, where $V$ is a finite dimensional vector space. Suppose that $X$ is some affine variety. A variety is an algebreo-geometric object that is determined by the vanishing of a finite number of polynomials $f_1, f_2, \cdots , f_k$. One can then consider the ring of regular functions on $X$: \[ k[X] \cong k[x_1, \ldots, x_n]/I(V) \] where the ideal $I(V)$ is determined by the polynomials $f_1, f_2, \cdots, f_k$ up to radical.
In 1900, Hilbert asked wheter or not certain types of rings are finitely generated. He was able to prove the following statement for the general linear group.
Hilbert's 14th Problem: If $G$ is a linear algebraic group, and $X$ an affine $G$-variety, is it true that the ring of invariants $k[X]^G$ is finitely generated? (Here $k$ is the complex numbers.)
It turns out that the answer to this question was in the negative. In the 1950's, Nagata found a counterexample. He gave a series of lectures on this topic, at the Tata Institute in 1965. The lecture notes can be found here.
So the statement, in general, was found to not hold. It does beg the question: With so many positive examples, does there exist a class of linear algebraic groups, for which Hilbert's 14th problem is true? The answer is yes! These linear algebraic groups are said to be linearly reductive ; meaning that their representations can be broken down into building blocks called reducible representations.
Note: It is perhaps natural to ask: Are there other types of linear algebraic groups beyond those that are linearly reductive? There are three notions that are interesting to this discussion:
The study of invariant theory lead to Hilber proving three major theorems:
David Mumford brought invariant theory back into the fold of modern research mathematics, in the 1960's. Harkening back to the way we started this discussion: let $G$ be some linear algebraic group and $X$ some $G$-variety. Perhaps the immediate question one could ask: Do the orbits of the action have the structure of a variety? It turns out that the answer is no. However, one can form the categorical quotient $X//G$, by utilizing the closed orbits of a group action on a variety. This construction lead to inquiries and exploaration around objects called moduli spaces that parameterize certain types of geometric objects.
Describing the ideas, in play, to study the ideas around categorical quotients causes difficulty. To even begin, one needs notions from algebraic geoemtry and the language of group actions. Beyond that, the theory is often quite subtle and technically difficult. An affine variety $V$ is a geometric object that can be described by the vanishing of a finite number of polynomials. A group $G$ is said to act regularly on $V$ if $G \times V \rightarrow V$ defined by $(g,v) \mapsto g \cdot v$ is a morphism of varieties. A group action partitions $V$ into a collection of orbits , where an orbit is: \[ \mathcal{O}_x = \{g \cdot x \mid g \in G\}. \] As stated, the "natural" question is:
Q: Can the orbits of the action be viewed as a variety?
A: Unfortunately, the answer to this is no.
This is only the beginning of the difficulty that arises. A continued study of GIT leads one through such topics as the Hilbert Nullcone, stability conditions and criteria, and moduli spaces. These are all very important concepts, both in modern mathematics and theoretcial physics.
Here is a set of notes to a talk I gave on the invariant theory of finite groups: Introduction to Representations and Invariant Theory
References:
Invariants of finite groups in characteristic $p$: Invariants of Finite Groups & Modular Representations
References:
Here is a set of notes that I prepared, for a talk I gave, while taking a course in Toric Varieties.
References: