> For the complete documentation index, see [llms.txt](https://pinpoint-apm.gitbook.io/pinpoint/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://pinpoint-apm.gitbook.io/pinpoint/documents/hbase-table-changes.md).

# HBase Table Changes

## 3.1.0

Table creation scripts for 3.1.0:

* [hbase-create.hbase](https://github.com/pinpoint-apm/pinpoint/blob/v3.1.0/hbase/scripts/hbase-create.hbase)
* [hbase-create-snappy.hbase](https://github.com/pinpoint-apm/pinpoint/blob/v3.1.0/hbase/scripts/hbase-create-snappy.hbase)

### Before You Start

Create the new HBase tables before changing table-related properties or starting upgraded components.

The property values shown below are the 3.1.0 defaults and use the new tables.

### Upgrade Order

1. Create the new HBase tables.
2. Enable dual write for each feature (see each feature's **Transition**).
3. Switch reads to the legacy tables during migration (see each feature's **Transition**).
4. Run the copy APIs to migrate historical data, if supported.
5. Switch reads back to the new tables, if changed in step 3.
6. Disable legacy writes once the new tables are confirmed healthy.

### Application & Agent List

#### Tables

`ApplicationIndex` is split into two new tables:

| Legacy             | New                      |
| ------------------ | ------------------------ |
| `ApplicationIndex` | `Application`, `AgentId` |

#### Properties

**Collector**

```properties
pinpoint.collector.application.index.v1.enabled=false
pinpoint.collector.application.index.v2.enabled=true
```

**Web**

```properties
# v1/v2 enabled properties are used for delete operations.
pinpoint.web.application.index.v1.enabled=false
pinpoint.web.application.index.v2.enabled=true
pinpoint.web.application.index.read.v2=true
```

**Batch**

```properties
pinpoint.batch.application.index.read.v2=true

# Legacy ApplicationIndex cleanup
job.cleanup.inactive.applications.enable=true
job.cleanup.inactive.applications.cron=0 0 3 * * THU
job.cleanup.inactive.applications.emptydurationthreshold=P35D
# - emptyItemWriter: Not actually removes application
# - cleanTargetWriter: Actually removes application
job.cleanup.inactive.applications.writer=emptyItemWriter

# New Application and AgentId table cleanup
# Disabled by default; enable after verifying with dry-run.
job.cleanup.inactive.agent-application.enable=false
job.cleanup.inactive.agent-application.cron=0 0 4 * * THU
# Test run only. Does not delete data.
job.cleanup.inactive.agent-application.dry-run=true
job.cleanup.inactive.agent-application.threshold-days=30
```

#### Transition

* Collector: set `pinpoint.collector.application.index.v1.enabled=true` for dual write.
* Web: set `pinpoint.web.application.index.v1.enabled=true` to delete from the legacy table during migration.
* Web: set `pinpoint.web.application.index.read.v2=false` to read from the legacy table during migration.
* Batch: set `pinpoint.batch.application.index.read.v2=false` to read from the legacy table during migration.

#### Data Migration

Application & Agent List supports historical data migration through Web copy APIs.

| API                                           | Description                                              |
| --------------------------------------------- | -------------------------------------------------------- |
| `/api/admin/copy/applications`                | Copy all application index data                          |
| `/api/admin/copy/agents/all`                  | Copy all agent index data, using more time and resources |
| `/api/admin/copy/agents?durationDays=30`      | Copy agent index data for recent `durationDays`          |
| `/api/admin/copy/agents?applicationName=app`  | Copy agent index data for a specific `applicationName`   |
| `/api/admin/copy/agents?serviceTypeCode=1000` | Copy agent index data for a specific `serviceTypeCode`   |

Notes:

* Copy APIs are exposed on the Web component.
* Agent copy APIs use the `AgentInfo` table; Java agents send `AgentInfo` periodically, so `durationDays` can include long-running active agents.
* Some agents may not send `AgentInfo` periodically; use `serviceTypeCode` for them.
* `durationDays` and `/agents/all` copy a wider range of data, so they may take longer to complete.
* `applicationName` and `serviceTypeCode` copy the least required data.

Example:

```bash
# copy all applications
curl -X POST /api/admin/copy/applications

# copy all agents
curl -X POST /api/admin/copy/agents/all

# copy agents for recent 30 days
curl -X POST "/api/admin/copy/agents?durationDays=30"

# copy agents for a specific application
curl -X POST "/api/admin/copy/agents?applicationName=my-app"

# copy agents for a specific service type
curl -X POST "/api/admin/copy/agents?serviceTypeCode=1000"
```

### ServerMap

#### Tables

`ApplicationMapStatisticsSelf_Ver2` is split into two new tables:

| Legacy                                | New                          |
| ------------------------------------- | ---------------------------- |
| `HostApplicationMap_Ver2`             | `MapAppHost`                 |
| `ApplicationMapStatisticsCaller_Ver2` | `MapAppOut`                  |
| `ApplicationMapStatisticsCallee_Ver2` | `MapAppIn`                   |
| `ApplicationMapStatisticsSelf_Ver2`   | `MapAppSelf`, `MapAgentSelf` |

#### Properties

**Collector**

```properties
# v2: legacy table, v3: new table, dual: dual write for data migration
pinpoint.modules.uid.version=v3
```

**Web**

```properties
# v2: legacy table, v3: new table
pinpoint.modules.uid.version=v3
```

#### Transition

* Collector: set `pinpoint.modules.uid.version=dual` for dual write.
* Web: set `pinpoint.modules.uid.version=v2` to read from the legacy tables during migration.

### TraceIndex

#### Tables

| Legacy                  | New          |
| ----------------------- | ------------ |
| `ApplicationTraceIndex` | `TraceIndex` |

#### Properties

**Collector**

```properties
pinpoint.collector.application.trace.index.v1.enabled=false
pinpoint.collector.application.trace.index.v2.enabled=true
```

**Web**

```properties
pinpoint.web.trace.index.read.v2=true
```

#### Transition

* Collector: set `pinpoint.collector.application.trace.index.v1.enabled=true` for dual write.
* Web: set `pinpoint.web.trace.index.read.v2=false` to read from the legacy table during migration.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://pinpoint-apm.gitbook.io/pinpoint/documents/hbase-table-changes.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.