PKCS11 (Public-Key Cryptography Standards), also known as “Cryptoki” or PKCS#11, is an API used to communicate with cryptographic security tokens such as smart cards, USB secure tokens, and Hardware Security Modules (HSMs).
HSM is always on the higher end in terms of cryptographic storage and for the cryptographic operations. HSM have the capability to store thousands or even more cryptographic keys and also have the implementation of large variety of cryptographic algorithms. HSM is normally considered as a server end product for storing the cryptographic keys centrally for corporate users and also for storing application specific keys and so on.
Smart card is a low end product in terms of cryptographic storage and also for the implementation of different cryptographic algorithms, which is fair as smart card belongs to a single user. Smart card may have the capability to store a single key for the user and also provide support for one or two cryptographic algorithms e.g. RSA for asymmetric key pair generation, signing and SHA256 algorithm for hashing.
The API defines the most commonly used cryptographic object types (RSA keys, X.509 Certificates, DES/Triple DES keys, etc.) and all the functions needed to use, create/generate, modify, and delete those objects.
PKCS11 is, at its core, an API used to create or delete cryptographic data like public-private key pairs.