com.yahoo.bard.webservice.metadata

Class DataSourceMetadataLoadTask

java.lang.Object

com.yahoo.bard.webservice.application.LoadTask<Boolean>

com.yahoo.bard.webservice.metadata.DataSourceMetadataLoadTask

All Implemented Interfaces:

Runnable

@Singleton
public class DataSourceMetadataLoadTask
extends LoadTask<Boolean>

Datasource metadata loader sends requests to the druid datasource metadata endpoint ('datasources') and returns the lists of available data segments for each datasource. It then builds Datasource Metadata objects which pivot this data into columns of intervals and then updates the DataSourceMetadataService.

Note that this uses the segmentMetadata query that touches the coordinator.

Nested Class Summary

Nested classes/interfaces inherited from class com.yahoo.bard.webservice.application.LoadTask

LoadTask.TaskFailureCallback

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATASOURCE_METADATA_QUERY_FORMAT
static String	DRUID_SEG_LOADER_TIMER_DELAY_KEY Parameter specifying the delay before the first run of the segment metadata loader, in milliseconds.
static String	DRUID_SEG_LOADER_TIMER_DURATION_KEY Parameter specifying the period of the segment metadata loader, in milliseconds.

Fields inherited from class com.yahoo.bard.webservice.application.LoadTask

delay, future, isPeriodic, loaderName, period

Constructor Summary

Constructors

Constructor and Description
DataSourceMetadataLoadTask(PhysicalTableDictionary physicalTableDictionary, DataSourceMetadataService metadataService, DruidWebService druidWebService, com.fasterxml.jackson.databind.ObjectMapper mapper) Datasource metadata loader fetches data from the druid coordinator and updates the datasource metadata service.

Method Summary

Modifier and Type	Method and Description
protected SuccessCallback	buildDataSourceMetadataSuccessCallback(DataSourceName dataSourceName) Callback to parse druid datasource metadata response.
protected SuccessCallback	buildDataSourceMetadataSuccessCallback(SingleDataSourcePhysicalTable table) Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable
protected HttpErrorCallback	getErrorCallback(DataSourceName dataSourceName) Get a default callback for an http error.
protected HttpErrorCallback	getErrorCallback(SingleDataSourcePhysicalTable table) Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable
org.joda.time.DateTime	getLastRunTimestamp() Return when this loader ran most recently.
protected void	queryDataSourceMetadata(DataSourceName dataSourceName) Queries Druid for updated datasource metadata and then updates the datasource metadata service.
protected void	queryDataSourceMetadata(SingleDataSourcePhysicalTable table) Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable
void	run()

Methods inherited from class com.yahoo.bard.webservice.application.LoadTask

getDefinedDelay, getDefinedPeriod, getErrorCallback, getFailureCallback, getFuture, getName, isPeriodic, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DATASOURCE_METADATA_QUERY_FORMAT

public static final String DATASOURCE_METADATA_QUERY_FORMAT

See Also:

Constant Field Values

DRUID_SEG_LOADER_TIMER_DURATION_KEY

public static final String DRUID_SEG_LOADER_TIMER_DURATION_KEY

Parameter specifying the period of the segment metadata loader, in milliseconds.

DRUID_SEG_LOADER_TIMER_DELAY_KEY

public static final String DRUID_SEG_LOADER_TIMER_DELAY_KEY

Parameter specifying the delay before the first run of the segment metadata loader, in milliseconds.

Constructor Detail

DataSourceMetadataLoadTask

public DataSourceMetadataLoadTask(PhysicalTableDictionary physicalTableDictionary,
                                  DataSourceMetadataService metadataService,
                                  DruidWebService druidWebService,
                                  com.fasterxml.jackson.databind.ObjectMapper mapper)

Datasource metadata loader fetches data from the druid coordinator and updates the datasource metadata service.

Parameters:

physicalTableDictionary - The physical tables with data sources to update

metadataService - The service that will store the metadata loaded by this loader

druidWebService - The druid webservice to query

mapper - Object mapper to parse druid metadata

Method Detail

run

public void run()

Specified by:

run in interface Runnable

Specified by:

run in class LoadTask<Boolean>

queryDataSourceMetadata

@Deprecated
protected void queryDataSourceMetadata(SingleDataSourcePhysicalTable table)

Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable

Queries Druid for updated datasource metadata and then updates the datasource metadata service.

Parameters:

table - The physical table to be updated.

queryDataSourceMetadata

protected void queryDataSourceMetadata(DataSourceName dataSourceName)

Queries Druid for updated datasource metadata and then updates the datasource metadata service.

Parameters:

dataSourceName - The data source to be updated.

buildDataSourceMetadataSuccessCallback

@Deprecated
protected final SuccessCallback buildDataSourceMetadataSuccessCallback(SingleDataSourcePhysicalTable table)

Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable

Callback to parse druid datasource metadata response.

Typical druid datasource metadata response:

  """
  {
      "name": "tableName",
      "properties": { },
      "segments": [
          {
              "dataSource": "tableName",
              "interval": "2015-01-01T00:00:00.000Z/2015-01-02T00:00:00.000Z",
              "version": "2015-01-15T18:08:20.435Z",
              "loadSpec": {
                  "type": "hdfs",
                  "path": "hdfs:/some_hdfs_URL/tableName/.../index.zip"
              },
              "dimensions": "color", "shape",
              "metrics": "height", "width",
              "shardSpec": {
                  "type":"hashed",
                  "partitionNum": 0,
                  "partitions": 2
              },
              "binaryVersion":9,
              "size":1024,
              "identifier":"tableName_2015-01-01T00:00:00.000Z_2015-01-02T00:00:00.000Z_2015-02-15T18:08:20.435Z"
          },
          {
              "dataSource": "tableName",
              "interval": "2015-01-01T00:00:00.000Z/2015-01-02T00:00:00.000Z",
              "version": "2015-02-01T07:02:05.912Z",
              "loadSpec": {
                  "type": "hdfs",
                  "path": "hdfs:/some_hdfs_URL/tableName/.../index.zip"
              },
              "dimensions": "color", "shape",
              "metrics": "height", "width",
              "shardSpec": {
                  "type":"hashed",
                  "partitionNum": 1,
                  "partitions": 2
              },
              "binaryVersion":9,
              "size":512,
              "identifier":"tableName_2015-01-01T00:00:00.000Z_2015-01-02T00:00:00.000Z_2015-02-01T07:02:05.912Z"
          }
      ]
   }"""
 

Parameters:

table - The table to inject into this callback.

Returns:

The callback itself.

buildDataSourceMetadataSuccessCallback

protected SuccessCallback buildDataSourceMetadataSuccessCallback(DataSourceName dataSourceName)

Callback to parse druid datasource metadata response.

Typical druid datasource metadata response:

  """
  {
      "name": "tableName",
      "properties": { },
      "segments": [
          {
              "dataSource": "tableName",
              "interval": "2015-01-01T00:00:00.000Z/2015-01-02T00:00:00.000Z",
              "version": "2015-01-15T18:08:20.435Z",
              "loadSpec": {
                  "type": "hdfs",
                  "path": "hdfs:/some_hdfs_URL/tableName/.../index.zip"
              },
              "dimensions": "color", "shape",
              "metrics": "height", "width",
              "shardSpec": {
                  "type":"hashed",
                  "partitionNum": 0,
                  "partitions": 2
              },
              "binaryVersion":9,
              "size":1024,
              "identifier":"tableName_2015-01-01T00:00:00.000Z_2015-01-02T00:00:00.000Z_2015-02-15T18:08:20.435Z"
          },
          {
              "dataSource": "tableName",
              "interval": "2015-01-01T00:00:00.000Z/2015-01-02T00:00:00.000Z",
              "version": "2015-02-01T07:02:05.912Z",
              "loadSpec": {
                  "type": "hdfs",
                  "path": "hdfs:/some_hdfs_URL/tableName/.../index.zip"
              },
              "dimensions": "color", "shape",
              "metrics": "height", "width",
              "shardSpec": {
                  "type":"hashed",
                  "partitionNum": 1,
                  "partitions": 2
              },
              "binaryVersion":9,
              "size":512,
              "identifier":"tableName_2015-01-01T00:00:00.000Z_2015-01-02T00:00:00.000Z_2015-02-01T07:02:05.912Z"
          }
      ]
   }"""
 

Parameters:

dataSourceName - The datasource name to inject into this callback.

Returns:

The callback itself.

getLastRunTimestamp

public org.joda.time.DateTime getLastRunTimestamp()

Return when this loader ran most recently.

Returns:

The date and time of the most recent execution of this loader.

getErrorCallback

@Deprecated
protected HttpErrorCallback getErrorCallback(SingleDataSourcePhysicalTable table)

Deprecated. Pass the DataSourceName directly, rather than via the PhysicalTable

Get a default callback for an http error.

Parameters:

table - The PhysicalTable that the error callback will relate to.

Returns:

A newly created http error callback object.

getErrorCallback

protected HttpErrorCallback getErrorCallback(DataSourceName dataSourceName)

Get a default callback for an http error.

Parameters:

dataSourceName - The data source that the error callback will relate to.

Returns:

A newly created http error callback object.