There are three common things in food which can make people sick: allergens, natural toxins, and spoilage.
Allergens must be substances recognized by our immune systems. For most people, they cause no trouble. But for a few people, the immune system can vastly over-react to some particular substance and flood the body with histamine. Histamine in small amounts is necessary for many body processes, but in larger amounts it is a poison, which causes rashes, vomiting, abdominal cramps and diarrhoea.
The human immune system is remarkable. There are millions of substances which it might need to recognize - the surface proteins of bacteria and viruses. But the body can't spare a gene for each of these proteins. There just aren't that many genes. Instead, the mammalian immune system uses only a few very complex genes. In the embryo stage, a large number of specialized immune system cells are produced and each of them makes a few random cuts in the DNA of its own copies of those genes and reassembles them. So, any one of the immune system cells has only a few genes devoted to recognizing a foreign substance, and can recognize only one substance, but there are many immune system cells and they can each recognize their own substance.
Then follows a training period. The immune cells that recognize the body's own proteins are permanently disabled. The others are kept around, but remember, for each substance there are only a few cells that can recognize it.
Now suppose a foreign substance gets into the blood stream for the first time. A few immune system cells find it and recognize it, but they are too few to combat it. Instead, they let other body defenses take responsibility, but those particular immune system cells reproduce, increasing their numbers so that the next time the substance is encountered they are ready for a massive response.
An allergy is a malfunction of this system. The substance recognized is a harmless food material, but the immune system treats it as it would an invading microbe, and it responds out of all proportion to the amount of allergen present.
So there are two characteristics of an allergic response. First, it must be something that you have been exposed to before. Second, your response doesn't depend much on the dose. A little is as bad as a lot.
If you suspect that you are allergic to something, an allergy specialist can check easily. A patch with a tiny quantity of the suspected allergen is placed on the skin and if there is a histamine reaction, you have found the allergen. You had best avoid coming in contact with it from then on.
The second and third ways food can cause you illness are due to toxins. There are toxins naturally present in the particular food, and there are toxins that don't belong there, caused by bacteria, fungi, etc.
Practically no food is entirely nutrients. Most plants produce poisons that help fight off insects or make the plant unpalatable to browsing animals. For example, celery contains psoralens, potatoes contain solanines and beans contain lectins (which are mostly destroyed by cooking - some kinds of raw beans are very toxic). Even something essential to life can be a poison if there's too much of it. The histamine which causes allergic reactions is the same histamine the body needs for normal life processes, but too much of it will produce the same effect whether it is an allergic reaction or a food ingredient. Most fish, cheese and wine contains some histamines.
The main difference between a toxin and an allergen is that a toxin has an effect proportional to the dose. If a milligram is bad, ten milligrams are ten times as bad.
Our bodies work to get rid of toxins. A few of them, like alcohol, can be transformed by special enzymes into harmless or even useful substances (which is why alcohol has calories). The liver is the organ specialized to transform toxins into useful substances. But if you overload the liver it will take a long time to detoxify and you will be sick until it has finished. Small amounts of these toxins will not be noticed.
With other toxins, the body's defense is to make the toxin water soluble and flush it out of the body as fast as possible. A particular protein called cytochrome P450 is a sort of detoxifying generalist. It can bind itself to a wide variety of small molecules and make them water soluble. The P450-toxin complex is then carried to the kidneys and becomes a component of urine.
Our bodies deal with toxins caused by spoilage in the same way. But, of course, they can be more dangerous because there can be rather a lot of them. They can also include some very nasty toxins indeed. We can process 10 milligrams of solanine and hardly notice it, but 0.000000008 milligrams of botulin will surely kill us.
Many people worry about residues from agricultural pesticides. In most cases these are toxins which the body can deal with exactly the way it deals with other toxins. Regulations regarding pesticide residues on food are meant to assure that the toxic load of pesticides is negligible in comparison with the natural toxin load present in the food.
Of course, any pesticide residue is worse than none at all. But it wouldn't make sense to avoid a tiny amount of pesticide residue in exchange for consuming a larger amount of some ``natural'' toxin.