Blockchain Indexers Explained: Everything You Need to Know

While blockchains have revolutionized how we think about data storage and verification, data stored onchain is inherently difficult to query, read, and use. Indexers solve all three problems.

In this guide, we’ll explain what blockchain indexers are and how they make blockchain data usable for applications and data analysts, as well as why they’re important for rollups. We’ll also look at a few notable indexer providers.

What is a blockchain indexer?

A blockchain indexer is a specialized tool that extracts transaction data from blockchain nodes, transforms it into a machine and human-readable form, and loads it into a database for easy querying. This makes indexers essential for developers who want to use blockchain data in their applications, investors or other observers who want to analyze what’s happening on a given blockchain, and other stakeholders. 

A blockchain is a continuously growing ledger of transactions. But the ledger is structured in a way that prioritizes security, immutability, and decentralization – not queryability. By default, blockchain data consists of a series of blocks, each containing a list of transactions linked cryptographically. This structure is best understood as an append-only linked list, which provides strong data integrity, but makes efficient data retrieval difficult. Trying to pull specific information from the blockchain under this structure would be like searching through thousands of pages of a ledger one by one.

Blockchain indexers solve the problem by organizing this raw data into structured formats that can be efficiently searched, filtered, and analyzed. Without indexers, the valuable data stored on blockchains would be largely inaccessible for practical use.

Indexer use cases: Who uses blockchain indexers and why?

Blockchain indexers are critical infrastructure for a variety of crypto use cases. We’ll look at a few examples of how different types of apps and services use indexers below.

Wallets

Crypto wallets use indexers to display transaction history, token balances, and NFT collections across multiple blockchains in their UIs. Indexers enable instant balance updates and transaction notifications without requiring wallets to scan the entire blockchain.

Analytics platforms

Data analytics services like Dune use indexing to transform raw blockchain data into tables that users can query and display visually, allowing them to create dashboards for crypto insights.

NFT marketplaces

NFT platforms rely on indexers to track ownership history, collection floor prices, trading volume, and other important pieces of information they need to display to users. 

DeFi applications

Indexers allow DeFi protocols to display users’ positions and other important information like trading trends and liquidity pool status in-app.

Security monitoring

Crypto security teams can use indexers to detect suspicious transactions, monitor smart contract vulnerabilities, and identify potential exploits in real-time.

Blockchain indexers for rollups

Indexers are also important partners for rollup teams for several reasons. 

For one, rollups need to attract developers to build on their chain – providing access to good indexing solutions so that builders can bring blockchain data into their apps is an important part of the developer experience. Indexers like Dune also allow rollups to make their data publicly available, so that analysts, investors, developers, and potential users can learn more about the chain. This is an important part of building interest in the rollup and encouraging adoption. Finally, rollup teams can also use indexing solutions for their own internal monitoring.

How do blockchain indexers work?

At a high level, indexers carry out an Extract, Transform, Load (ETL) process commonly used for web2 data analytics, but with blockchain data. We’ll break down the steps below.

Data extraction

The first step in the indexing process involves connecting to blockchain nodes to extract raw data. Indexers connect to nodes via RPC (Remote Procedure Call) endpoints and listen for new blocks and transactions in real time. They then extract data as these events occur, including:

• Transaction details (sender, receiver, amount, timestamp)
• Smart contract events and function calls
• Block headers and metadata
• Account state changes

Indexer services can extract this data by running their own full node to connect to, or by connecting to third-party node providers via APIs.

Data transformation

Once the indexer has extracted raw blockchain data, it transforms it into structured formats that are easier to work with. This process involves:

• Parsing and normalizing raw blockchain data into consistent formats
• Decoding smart contract events according to their Application Binary Interfaces (ABIs)
• Linking together related data points (e.g., connecting transactions to their respective blocks)
• Enriching data with additional context (e.g., token prices at the time of transactions)

This transformation process is crucial because blockchain data in its native form isn’t encoded in a way that’s conducive to database storage or readability. 

Data storage and retrieval

After transformation, the processed data is stored in databases optimized for quick retrieval. 

Most indexers use relational databases like PostgreSQL or NoSQL databases like MongoDB. The indexer will then organize the data into tables or collections with appropriate indexes, and apply query optimization techniques to ensure fast response times. On the developer-facing end, indexers provide APIs to allow applications to access the indexed, organized data tables. 

This three-step process ensures that blockchain data can be accessed in milliseconds, allowing onchain applications to use the data in real time.

Components of a blockchain indexer

Blockchain indexers consist of five key components, which we’ll outline below.

Blockchain node access point

Indexers first need a way of accessing a node on the blockchain in order to extract the raw data. This typically comes in one of three ways:

• A full node that maintains a complete copy of the blockchain
• A light client that verifies only the data it needs
• A third-party node service that provides API access

For example, an Ethereum indexer might connect to an Ethereum node via JSON-RPC to retrieve block and transaction data.

Database

The database stores the extracted blockchain data in a structured format. Different indexers use different database technologies based on their specific requirements:

• Relational databases like PostgreSQL offer robust data modeling with powerful query capabilities
• NoSQL databases like MongoDB provide flexibility for storing varied data structures
• Specialized time-series databases like InfluxDB can be used for historical analysis

The choice of database will depend on the specific use case of the indexer in question.

Indexing engine

The indexing engine is the core component that processes blockchain data and extracts relevant information. It handles:

• Block and transaction parsing
• Smart contract event decoding
• Data normalization and enrichment
• Index creation and maintenance

The indexing engine is what transforms the data into something readable and usable.

Search API

The search API provides interfaces for users and applications to query the indexed data. This can include:

• RESTful endpoints for common queries
• GraphQL interfaces for more complex, custom-built queries
• WebSocket connections for real-time data streaming

Search APIs allow developers to pull the blockchain data they need according to specific parameters like addresses, transaction IDs, block numbers, and more.

User interface

While not all indexers include a user interface (UI), many provide web-based dashboards that allow users to:

• Explore indexed data visually
• Build and test queries
• Monitor indexer performance
• Configure indexing parameters

Types of blockchain indexers and indexer examples

Blockchain indexers come in different varieties to serve specific use cases. We’ll run through the primary types of blockchain indexers below.

General purpose indexers

General-purpose indexers are designed to index a wide range of blockchain data without focusing on specific use cases. They typically:

• Index all transactions and events on supported blockchains
• Provide flexible query capabilities for diverse applications
• Support multiple blockchain networks

Goldsky is one example of a general purpose indexer. It supports over 90 blockchains and typically indexes all onchain data for those blockchains, enabling a broad range of developer and analytics use cases.

Etherscan is a good example of a general purpose indexer focused on one specific chain: Ethereum. Etherscan collects all transaction activity on Ethereum and stores it indefinitely, allowing for historical analysis, and also has a suite of APIs for developer use cases. 

NFT indexers

As the name would suggest, NFT indexers are designed for querying data related to NFTs, such as:

• Token metadata and ownership history
• Marketplace activities (listings, sales, bids)
• Collection statistics and floor prices

These indexers enable developers to build NFT marketplaces, wallet functionalities, and analytics platforms. One of the most popular NFT indexers is NFTScan, which offers NFT data access across over a dozen chains, and also hosts NFT-specific block explorers for those chains.

Decentralized indexers

While most indexers are owned and operated by a single entity or team, decentralized indexers allow anyone to become part of their network of indexers by joining permissionlessly. In keeping with the general crypto ethos around decentralization, this framework can reduce the risk of a centralized point of failure in the indexing solution.

The Graph is a highly popular decentralized indexer protocol, and provides a good example of how decentralized indexers typically work:

• A network of independent node operators make up The Graph’s network and compete to provide indexing services and process queries.
• Indexers stake The Graph’s GRT token, incentivizing them to provide accurate data – otherwise, their staked GRT is slashed.
• Users can also pay for The Graph’s indexing services using GRT

Blockchain indexers are a core part of the onchain application tech stack

Indexers solve a fundamental blockchain problem of making blockchain data usable. The right indexer solution will vary by project – developers must judge their options carefully depending on their use case.

Rollup teams also need to partner with strong indexer providers in order to attract developers to their chain given how crucial indexers are to app development. Depending on the chain’s use case, most rollups should seek to get indexed by a mix of different providers to ensure the best possible experience for all developers they want to attract. 

Looking for indexing partners for your rollup? Check out Conduit Marketplace. No provider makes it easier for developers to integrate with a variety of indexers and data platforms providing indexing services.