Tokenisation (AI)

Definition

The process of breaking text, code, or other sequential data into discrete units (tokens) that serve as the input and output elements for large language models. Tokenisation determines how a model processes language and directly affects inference costs, since API pricing for large language models is typically based on token count. Different tokenisation schemes handle multilingual content with varying efficiency.

Complementary Terms

Concepts that frequently appear alongside Tokenisation (AI) in practice.

Large Language Model

A type of neural network trained on vast corpora of text data, capable of generating human-like text, answering questions, summarising documents, and performing reasoning tasks. Large language models such as GPT and Claude represent significant R&D investment and are reshaping knowledge work, customer service, and content production across industries.

Reporting Unit

The level at which goodwill is tested for impairment under US GAAP (ASC 350), defined as an operating segment or one level below an operating segment (a component). A component is a reporting unit if it constitutes a business for which discrete financial information is available and segment management regularly reviews its operating results.

Inference Cost

The computational expense of running a trained AI model to generate predictions or outputs in production. Inference costs directly impact the unit economics of AI-powered products and services, and are a key consideration in pricing, margin analysis, and the financial viability of AI deployments at scale.

Labour Productivity

The amount of output produced per unit of labour input, commonly measured as gross value added (GVA) divided by labour costs or number of employees. Labour productivity is a key efficiency metric that reflects the quality of human capital, processes, and technology deployed by a firm.

Generative AI

A category of artificial intelligence systems capable of creating new content — including text, images, code, music, and video — based on patterns learned from training data. Generative AI is transforming content production, product design, and software development, raising novel questions about intellectual property ownership and the valuation of AI-generated outputs.

Prompt Engineering

The practice of designing and optimising input instructions (prompts) to elicit desired outputs from large language models and other generative AI systems. Effective prompt engineering can significantly improve AI output quality and consistency, and documented prompt libraries are emerging as a form of organisational knowledge capital.

Retrieval-Augmented Generation (RAG) Architecture

A technical architecture that enhances large language model outputs by retrieving relevant information from an external knowledge base before generating a response, grounding the model's output in verified, up-to-date, and domain-specific data. RAG reduces hallucination risk, enables LLMs to access proprietary or recent information not in their training data, and provides citation capabilities.

Data Pipeline

An automated sequence of data processing steps that extracts, transforms, and loads data from source systems into target systems for analysis, reporting, or machine learning model training. Well-architected data pipelines are critical infrastructure assets that enable data-driven decision-making and AI deployment, and their reliability directly impacts downstream business processes.

Put this knowledge to work

Use Opagio's free tools to measure and grow the intangible assets that drive your business value.