What is Self-Supervised Learning?

The word self-supervised learning is a means for training computers to do tasks without humans providing labeled data. Self-supervised learning can also be an autonomous form of supervised learning because it does not require human input in the form of data labeling.

The term self-supervised learning has been widely used to refer to techniques that do not use human-annotated datasets to learn (visual) representations of the data (i.e. representation learning). The idea behind self-supervised learning is to develop a deep learning system that can learn to fill in the blanks.

The closest we have to self-supervised learning systems are Transformers, an architecture that has proven very successful in natural language processing (NLP). Transformers don’t require labeled data. They are trained on large corpora of unstructured text such as Wikipedia articles. And they’ve proven to be much better than their predecessors at generating text, engaging in conversation, and answering questions. (But they are still very far from really understanding human language.)

Transformers have become very popular and are the underlying technology for nearly all state-of-the-art language models, including Google’s BERT, Facebook’s RoBERTa, Open AI’s GPT2, and Google’s Meena chatbot.

More recently, Artificial Intelligence (AI) researchers have proven that transformers can perform integration and solve differential equations, problems that require symbol manipulation. This might be a hint that the evolution of transformers might enable neural networks to move beyond pattern recognition and statistical approximation tasks.

E-SPIN as being a value integrator to assist enterprise customers to implement various digital transformation technology, including learning management system (LMS), e-learning system and portal, include self-supervised learning machine to accelerate their speed, scale and scope objective in related to digital transformation. From e-learning system and portal infrastructure, application and performance monitoring, continuous security protection E-SPIN established in 2005, in the business of supply, consultancy, integration, training and maintenance of established and emerging technology and systems for enterprise customers and government agencies. Feel free to contact E-SPIN for your project and operation requirements.