Tokenisation (AI)

Definition

The process of breaking text, code, or other sequential data into discrete units (tokens) that serve as the input and output elements for large language models. Tokenisation determines how a model processes language and directly affects inference costs, since API pricing for large language models is typically based on token count. Different tokenisation schemes handle multilingual content with varying efficiency.

Complementary Terms

Concepts that frequently appear alongside Tokenisation (AI) in practice.

Large Language Model

A type of neural network trained on vast corpora of text data, capable of generating human-like text, answering questions, summarising documents, and performing reasoning tasks. Large language models such as GPT and Claude represent significant R&D investment and are reshaping knowledge work, customer service, and content production across industries.

Reporting Unit

The level at which goodwill is tested for impairment under US GAAP (ASC 350), defined as an operating segment or one level below an operating segment (a component). A component is a reporting unit if it constitutes a business for which discrete financial information is available and segment management regularly reviews its operating results.

Inference Cost

The computational expense of running a trained AI model to generate predictions or outputs in production. Inference costs directly impact the unit economics of AI-powered products and services, and are a key consideration in pricing, margin analysis, and the financial viability of AI deployments at scale.

Generative AI

A category of artificial intelligence systems capable of creating new content — including text, images, code, music, and video — based on patterns learned from training data. Generative AI is transforming content production, product design, and software development, raising novel questions about intellectual property ownership and the valuation of AI-generated outputs.

Prompt Engineering

The practice of designing and optimising input instructions (prompts) to elicit desired outputs from large language models and other generative AI systems. Effective prompt engineering can significantly improve AI output quality and consistency, and documented prompt libraries are emerging as a form of organisational knowledge capital.

Retrieval-Augmented Generation (RAG) Architecture

A technical architecture that enhances large language model outputs by retrieving relevant information from an external knowledge base before generating a response, grounding the model's output in verified, up-to-date, and domain-specific data. RAG reduces hallucination risk, enables LLMs to access proprietary or recent information not in their training data, and provides citation capabilities.

Data Pipeline

An automated sequence of data processing steps that extracts, transforms, and loads data from source systems into target systems for analysis, reporting, or machine learning model training. Well-architected data pipelines are critical infrastructure assets that enable data-driven decision-making and AI deployment, and their reliability directly impacts downstream business processes.

Non-Fungible Token (NFT)

A unique cryptographic token recorded on a blockchain that represents ownership of a specific digital or physical asset, such as artwork, music, collectibles, or virtual real estate. Unlike fungible tokens (such as cryptocurrencies), each NFT is distinct and cannot be exchanged on a one-for-one basis with another.

Further Reading

The Productivity Measurement Gap: Why GDP Per Hour Worked No Longer Tells the Full Story

Traditional productivity metrics were designed for an economy dominated by physical capital and tangible output. In a world where the most valuable firms derive their competitive advantage from software, data, brand equity, and organizational know-how, those metrics are increasingly unreliable. Here is why the gap matters — and what we should do about it.

Read more →

Put this knowledge to work

Use Opagio's free tools to measure and grow the intangible assets that drive your business value.