Plugin Developer Guide
You can write Pinpoint profiler plugins to extend profiling target coverage. It is highly advisable to look into the trace data recorded by pinpoint plugins before jumping into plugin development.
- There is a fast auto pinpoint agent plugin generate tool from a 3rd party for creating a simple plug-in, if you'd like to check out.
In Pinpoint, a transaction consists of a group of
Spanrepresents a trace of a single logical node where the transaction has gone through.
To aid in visualization, let's suppose that there is a system like below. The FrontEnd server receives requests from users, then sends request to the BackEnd server, which queries a DB. Among these nodes, let's assume only the FrontEnd and BackEnd servers are profiled by the Pinpoint Agent.
When a request arrives at the FrontEnd server, Pinpoint Agent generates a new transaction id and creates a
Spanwith it. To handle the request, the FrontEnd server then invokes the BackEnd server. At this point, Pinpoint Agent injects the transaction id (plus a few other values for propagation) into the invocation message. When the BackEnd server receives this message, it extracts the transaction id (and the other values) from the message and creates a new
Spanwith them. Resulting, all
Spansin a single transaction share the same transaction id.
Spanrecords important method invocations and their related data(arguments, return value, etc) before encapsulating them as
SpanEventsin a call stack like representation. The
Spanitself and each of its
SpanEventsrepresents a method invocation.
SpanEventhave many fields, but most of them are handled internally by Pinpoint Agent and most plugin developers won't need to worry about them. But the fields and data that must be handled by plugin developers will be listed throughout this guide.
Pinpoint plugin consists of type-provider.yml and
ProfilerPluginimplementations. type-provider.yml defines the
AnnotationKeysthat will be provided by the plugin, and provides them to Pinpoint Agent, Web and Collector.
ProfilerPluginimplementations are used by Pinpoint Agent to transform target classes to record trace data.
Plugins are deployed as jar files. These jar files are packaged under the plugin directory for the agent, while the collector and web have them deployed under WEB-INF/lib. On start up, Pinpoint Agent, Collector, and Web iterates through each of these plugins; parses type-provider.yml, and loads
ServiceLoaderfrom the following locations:
type-provider.yml defines the
AnnotationKeysthat will be used by the plugin and provided to the agent, collector and web; the format of which is outlined below.
- code: <short>
desc: <String> # May be omitted, defaulting to the same value as name.
property: # May be omitted, all properties defaulting to false.
terminal: <boolean> # May be omitted, defaulting to false.
queue: <boolean> # May be omitted, defaulting to false.
recordStatistics: <boolean> # May be omitted, defaulting to false.
includeDestinationId: <boolean> # May be omitted, defaulting to false.
alias: <boolean> # May be omitted, defaulting to false.
matcher: # May be omitted
type: <String> # Any one of 'args', 'exact', 'none'
code: <int> # Annotation key code - required only if type is 'exact'
- code: <int>
property: # May be omitted, all properties defaulting to false.
AnnotationKeydefined here are instantiated when the agent loads, and can be obtained using
ServiceType serviceType = ServiceTypeProvider.getByCode(1000); // by ServiceType code
ServiceType serviceType = ServiceTypeProvider.getByName("NAME"); // by ServiceType name
AnnotationKey annotationKey = AnnotationKeyProvider.getByCode("100");
ServiceTyperepresents which library the traced method belongs to, as well as how the
SpanEventshould be handled.
The table below shows the
name of the
short type code value of the
ServiceTypecode must use a value from its appropriate category. The table below shows these categories and their range of codes.
0 ~ 999
1000 ~ 1999
2000 ~ 2999
8000 ~ 8999
9000 ~ 9999
5000 ~ 7999
ServiceTypecode must be unique. Therefore, if you are writing a plugin that will be shared publicly, you must contact Pinpoint dev. team to get a
ServiceTypecode assigned. If your plugin is for private use, you may freely pick a value for
ServiceTypecode from the table below.
1900 ~ 1999
2900 ~ 2999
8900 ~ 8999
9900 ~ 9999
7500 ~ 7999
ServiceTypescan have the following properties.
Pinpoint Collector should collect execution time statistics of this
The service may or may not have Pinpoint-Agent attached at the following service but regardlessly have knowledge what will follow. (Ex. Elasticsearch client)
You can annotate spans and span events with more information. An Annotation is a key-value pair where the key is an
AnnotationKeytype and the value is a primitive type, String or a byte. There are pre-defined
AnnotationKeysfor commonly used annotation types, but you can define your own keys in type-provider.yml if these are not enough.
Name of the
int type code value of the
If you are writing a plugin for public use, and are looking to add a new
AnnotationKey, you must contact the Pinpoint dev. team to get an
AnnotationKeycode assigned. If your plugin is for private use, you may pick a value between 900 to 999 safely to use as
The table below shows the
Show this annotation in transaction call tree.
This property is not for plugins.
You may also define and attach an
matcherelement in the sample type-provider code above). If you attach an
AnnotationKeyMatcherthis way, matching annotations will be displayed as representative annotation when the
SpanEventis displayed in the transaction call tree.
ProfilerPluginmodifies target library classes to collect trace data.
ProfilerPluginworks in the order of following steps:
- 1.Pinpoint Agent is started when the JVM starts.
- 2.Pinpoint Agent loads all plugins under
- 3.Pinpoint Agent invokes
ProfilerPlugin.setup(ProfilerPluginSetupContext)for each loaded plugin.
- 4.In the
setupmethod, the plugin registers a
TransformerCallbackto all classes that are going to be transformed.
- 5.Target application starts.
- 6.Every time a class is loaded, Pinpoint Agent looks for the
TransformerCallbackregistered to the class.
- 7.If a
TransformerCallbackis registered, the Agent invokes it's
TransformerCallbackmodifies the target class' byte code. (e.g. add interceptors, add fields, etc.)
- 9.The modified byte code is returned to the JVM, and the class is loaded with the returned byte code.
- 10.Application continues running.
- 11.When a modified method is invoked, the injected interceptor's
aftermethods are invoked.
- 12.The interceptor records the trace data.
The most important points to consider when writing a plugin are 1) figuring out which methods are interesting enough to warrant tracing, and 2) injecting interceptors to actually trace these methods. These interceptors are used to extract, store, and pass trace data around before they are sent off to the Collector. Interceptors may even cooperate with each other, sharing context between them. Plugins may also aid in tracing by adding getters or even custom fields to the target class so that the interceptors may access them during execution. Pinpoint plugin sample shows you how the
TransformerCallbackmodifies classes and what the injected interceptors do to trace a method.
We will now describe what interceptors must do to trace different kinds of methods.
Plain method refers to anything that is not a top-level method of a node, or is not related to remote or asynchronous invocation. Sample 2 shows you how to trace these plain methods.
Top level method of a node is a method in which its interceptor begins a new trace in a node. These methods are typically acceptors for RPCs, and the trace is recorded as a
ServiceTypecategorized as a server.
Spanis recorded depends on whether the transaction has already begun at any previous nodes.
2.2.1 New transaction
If the current node is the first one that is recording the transaction, you must issue a new transaction id and record it.
TraceContext.newTraceObject()will handle this task automatically, so you will simply need to invoke it.
2.2.2 Continue Transaction
If the request came from another node traced by a Pinpoint Agent, then the transaction will already have a transaction id issued; and you will have to record the data below to the
Span. (Most of these data are sent from the previous node, usually packed in the request message)
Span ID of the previous node
Application name of the previous node
Application type of the previous node
Procedure name (Optional)
Server(current node) address
Server address that the client used
Pinpoint finds caller-callee relation between nodes using acceptorHost. In most cases, acceptorHost is identical to endPoint. However, the address which client sent the request to may sometimes be different from the address the server received the request (proxy). To handle such cases, you have to record the actual address the client used to send the request to as acceptorHost. Normally, the client plugin will have added this address into the request message along with the transaction data.
Moreover, you must also use the span id issued and sent by the previous node.
Sometimes, the previous node marks the transaction to not be traced. In this case, you must not trace the transaction.
As you can see, the client plugin must be able pass trace data to the server plugin, and how to do this is protocol dependent.
An interceptor of a method that invokes a remote node has to record the following data:
Target server address
Logical name of the target
Invoking target procedure name (optional)
Span id that will be used by next node's span (If next node is traceable by Pinpoint)
Whether or not the next node is traceable by Pinpoint affects how the interceptor is implemented. The term "traceable" here is about possibility. For example, a HTTP client's next node is a HTTP server. Pinpoint does not trace all HTTP servers, but it is possible to trace them (and there already are HTTP server plugins). In this case, the HTTP client's next node is traceable. On the other hand, MySQL JDBC's next node, a MySQL database server, is not traceable.
2.3.1 If the next node is traceable
If the next node is traceable, the interceptor must propagate the following data to the next node. How to pass them is protocol dependent, and in worst cases may be impossible to pass them at all.
Application name of current node
Application type of current node
Span id of trace at current node
Span id that will be used by the next node's span (same value with nextSpanId of above table)
Pinpoint finds out caller-callee relation by matching destinationId of client trace and acceptorHost of server trace. Therefore the client plugin has to record destinationId and the server plugin has to record acceptorHost with the same value. If server cannot acquire the value by itself, client plugin has to pass it to server.
The interceptor's recorded
ServiceTypemust be from the RPC client category.
2.3.2 If the next node is not traceable
If the next node is not traceable, your
ServiceTypemust have the
If you want to record the destinationId, it must also have the
INCLUDE_DESTINATION_IDproperty. If you record destinationId, server map will show a node per destinationId even if they have same endPoint.
ServiceTypemust be a DB client or Cache client category. Note that you do not need to concern yourself about the terms "DB" or "Cache", as any plugin tracing a client library with non-traceable target server may use them. The only difference between "DB" and "Cache" is the time range of the response time histogram ("Cache" having smaller intervals for the histogram).
Trace objects are bound to the thread that first created them via ThreadLocal and whenever the execution crosses a thread boundary, trace objects are lost to the new thread. Therefore, in order to trace tasks across thread boundaries, you must take care of passing the current trace context over to the new thread. This is done by injecting an AsyncContext into an object shared by both the invocation thread and the execution thread. The invocation thread creates an AsyncContext from the current trace, and injects it into an object that will be passed over to the execution thread. The execution thread then retrieves the AsyncContext from the object, creates a new trace out of it and binds it to it's own ThreadLocal. You must therefore create interceptors for two methods : i) one that initiates the task (invocation thread), and ii) the other that actually handles the task (execution thread).
The initiating method's interceptor has to issue an AsyncContext and pass it to the handling method. How to pass this value depends on the target library. In worst cases, you may not be able to pass it at all.
The handling method's interceptor must then continue the trace using the propagated AsyncContext and bind it to it's own thread. However, it is very strongly recommended that you simply extend the AsyncContextSpanEventSimpleAroundInterceptor so that you do not have to handle this manually.
Keep in mind that since the shared object must be able have AsyncContext injected into it, you have to add a field using
AsyncContextAccessorduring it's class transformation. You can find an example for tracing asynchronous tasks here.
HTTP client is an example of a method invoking a remote node (client), and HTTP server is an example of a top level method of a node (server). As mentioned before, client plugins must have a way to pass transaction data to server plugins to continue the trace. Note that the implementation is protocol dependent, and HttpMethodBaseExecuteMethodInterceptor of HttpClient3 plugin and StandardHostValveInvokeInterceptor of Tomcat plugin show a working example of this for HTTP:
- 2.Client plugin records
IP:PORTof the server as
- 3.Client plugin passes
destinationIdvalue to server as
- 4.Server plugin records
Header.HTTP_HOSTheader value as
One more thing you have to remember is that all the clients and servers using the same protocol must pass the transaction data in the same way to ensure compatibility. So if you are writing a plugin of some other HTTP client or server, your plugin has to record and pass transaction data as described above.
You can run plugin integration tests (
mvn integration-test) with PinointPluginTestSuite, which is a JUnit Runner. It downloads all the required dependencies from maven repositories and launches a new JVM with the Pinpoint Agent and the aforementioned dependencies. The JUnit tests are executed in this JVM.
To run the plugin integration test, it needs a complete agent distribution - which is why integration tests are in the plugin-sample-agent module and why they are run in integration-test phase.
For the actual integration test, you will want to first invoke the method you are tracing, and then use PluginTestVerifier to check if the trace data is correctly recorded.
PinointPluginTestSuitedoesn't use the project's dependencies (configured in pom.xml). It uses the dependencies that are listed by
@Dependencyannotation. This way, you may test multiple versions of the target library using the same test class.
Dependencies are declared as following. You may specify versions or version ranges for a dependency library.
PinointPluginTestSuiteby default searches the local repository and maven central repository. You may also add your own repositories by using the
You can specify the JVM version for a test using
@JvmVersionis not present, JVM at
java.home propertywill be used.
PinpointPluginTestSuiteis not for applications that has to be launched by its own main class. You can extend AbstractPinpointPluginTestSuite and related types to test such applications.
If you're developing a plugin for applications, you need to add images so the server map can render the corresponding node. The plugin jar itself cannot provide these image files and for now, you will have to add the image files to the web module manually.
First, put the PNG files to following directories:
- web/src/main/webapp/images/icons (25x25)
- web/src/main/webapp/images/servermap (80x40)
ServiceTypename and the image file name to