Migrate to Transaction Stream Service

This guide contains information on how to migrate to using the Transaction Stream Service if you are currently running a legacy indexer.

The old indexer stack requires running an archival fullnode with additional threads to process the transactions which is difficult and expensive to maintain. Adding more custom logic either requires a bulkier machine, or running several fullnodes that scale linearly.

This new way of indexing uses the Transaction Stream Service. You can either use the Labs-Hosted Transaction Stream Service or run your own instance of Transaction Stream Service.

# SSH
git clone git@github.com:aptos-labs/aptos-indexer-processors.git

# HTTPS
git clone https://github.com/aptos-labs/aptos-indexer-processors.git

Navigate to the directory for the service:

cd aptos-indexer-processors
cd rust/processor

For each processor you’re migrating, you’ll need to create a config file using the template below. You can find more information about each field of the config file here.

health_check_port: 8084
server_config:
  processor_config:
    type: default_processor
  postgres_connection_string: <postgres_uri, e.g. postgresql://postgres:@localhost:5432/indexer>
  indexer_grpc_data_service_address: <url_from_api_gateway>
  indexer_grpc_http2_ping_interval_in_secs: 60
  indexer_grpc_http2_ping_timeout_in_secs: 10
  auth_token: <auto_token_from_api_gateway>
  starting_version: 0 # optional
  ending_version: 0 # optional

To connect the processor to the Transaction Stream Service, you need to set the URL for indexer_grpc_data_service_address. Choose one of the following options.

The main benefit of using the Labs-Hosted Transaction Stream Service is that you no longer need to run an archival fullnode to get a stream of transactions. This service is rate-limited. Instructions to connect to Labs-Hosted Transaction Stream can be found here.

If you choose to, you can run a self-hosted instance of the Transaction Stream Service and connect your processors to it. Instructions to run a Self-Hosted Transaction Stream can be found here.

If you have custom processors written with the old indexer, we highly recommend starting from scratch with a new database. Using a new database ensures that all your custom database migrations will be applied during this migration.

Migrate your custom schemas by copying over each of your custom migrations to the migrations folder.

Migrate the code by copying over your custom processors to the processors folder and any relevant custom models to the models folder. Integrate the custom processors with the rest of the code by adding them to the following Rust code files.

mod.rs

pub enum Processor {
    ...
    CoinProcessor,
    ...
}

impl Processor {
    ...
    COIN_PROCESSOR_NAME => Self::CoinProcessor,
    ...
}

worker.rs

Processor::CoinProcessor => {
    Arc::new(CoinTransactionProcessor::new(self.db_pool.clone()))
},

Even though the new processors have the same Postgres schemas as the old ones, we recommend you do a complete backfill (ideally writing to a new DB altogether) because some fields are a bit different as a result of the protobuf conversion.

These instructions assume you are familiar with using Diesel migrations. Run the full database migration with the following command:

DATABASE_URL=postgres://postgres@localhost:5432/postgres diesel migration run

To run a single processor, use the following command:

cargo run --release -- -c config.yaml

If you have multiple processors, you’ll need to run a separate instance of the service for each of the processors.

If you’d like to run the processor as a Docker image, the instructions are listed here.

The protobuf schema may be updated in the future. Backwards incompatible changes will be communicated in release notes.

Since the new indexer stack has the same Postgres schema as the old indexer stack, it should be easy to migrate your processors. We still highly recommend creating a new DB for this migration so that any custom DB migrations are applied.

Follow Step 3 in this guide to migrate your custom logic over to the new processors stack.