com.yahoo.bard.webservice.web.responseprocessors

Class DruidPartialDataResponseProcessor

java.lang.Object

com.yahoo.bard.webservice.web.responseprocessors.DruidPartialDataResponseProcessor

All Implemented Interfaces:

FullResponseProcessor, ResponseProcessor

public class DruidPartialDataResponseProcessor
extends Object
implements FullResponseProcessor

Response processor for finding missing partial data in Druid

In druid version 0.9.0 or later, druid implemented a feature that returns missing intervals for a given query. For example

 
 Content-Type: application/json
 200 OK
 Date:  Mon, 10 Apr 2017 16:24:24 GMT
 Content-Type:  application/json
 X-Druid-Query-Id:  92c81bed-d9e6-4242-836b-0fcd1efdee9e
 X-Druid-Response-Context: {
     "uncoveredIntervals": [
         "2016-11-22T00:00:00.000Z/2016-12-18T00:00:00.000Z","2016-12-25T00:00:00.000Z/2017-
         01-03T00:00:00.000Z","2017-01-31T00:00:00.000Z/2017-02-01T00:00:00.000Z","2017-02-
         08T00:00:00.000Z/2017-02-09T00:00:00.000Z","2017-02-10T00:00:00.000Z/2017-02-
         13T00:00:00.000Z","2017-02-16T00:00:00.000Z/2017-02-20T00:00:00.000Z","2017-02-
         22T00:00:00.000Z/2017-02-25T00:00:00.000Z","2017-02-26T00:00:00.000Z/2017-03-
         01T00:00:00.000Z","2017-03-04T00:00:00.000Z/2017-03-05T00:00:00.000Z","2017-03-
         08T00:00:00.000Z/2017-03-09T00:00:00.000Z"
     ],
     "uncoveredIntervalsOverflowed": true
 }
 Content-Encoding:  gzip
 Vary:  Accept-Encoding, User-Agent
 Transfer-Encoding:  chunked
 Server:  Jetty(9.2.5.v20141112)
 
 

The missing intervals are indicated in "uncoveredIntervals". We compare it to the missing intervals that we expects from Partial Data V1. If "uncoveredIntervals" contains any interval that is not present in our expected missing interval list, we can send back an error response indicating the mismatch in data availability before the response is cached.

Constructor Summary

Constructors

Constructor and Description
DruidPartialDataResponseProcessor(ResponseProcessor next) Constructor.

Method Summary

Modifier and Type	Method and Description
HttpErrorCallback	getErrorCallback(DruidAggregationQuery<?> druidQuery) Callback for handling http errors.
FailureCallback	getFailureCallback(DruidAggregationQuery<?> druidQuery) Callback handler for unexpected failures.
ResponseContext	getResponseContext() The response context allows state to be injected from construction and visible across response processor layers as necessary.
void	processResponse(com.fasterxml.jackson.databind.JsonNode json, DruidAggregationQuery<?> query, LoggingContext metadata) If status code is 200, do the following Extract uncoveredIntervalsOverflowed from X-Druid-Response-Context inside the JsonNode passed into DruidPartialDataResponseProcessor::processResponse, if it is true, invoke error response saying limit overflowed, Extract uncoveredIntervals from X-Druid-Response-Contex inside the JsonNode passed into DruidPartialDataResponseProcessor::processResponse, Parse both the uncoveredIntervals extracted above and allAvailableIntervals extracted from the union of all the query's datasource's availabilities from DataSourceMetadataService into SimplifiedIntervalLists, Compare both SimplifiedIntervalLists above, if allAvailableIntervals has any overlap with uncoveredIntervals, invoke error response indicating druid is missing some data that are we are expecting to exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DruidPartialDataResponseProcessor

public DruidPartialDataResponseProcessor(ResponseProcessor next)

Constructor.

Parameters:

next - Next ResponseProcessor in the chain

Method Detail

getResponseContext

public ResponseContext getResponseContext()

Description copied from interface: ResponseProcessor

The response context allows state to be injected from construction and visible across response processor layers as necessary.

Specified by:

getResponseContext in interface ResponseProcessor

Returns:

The context data

getFailureCallback

public FailureCallback getFailureCallback(DruidAggregationQuery<?> druidQuery)

Description copied from interface: ResponseProcessor

Callback handler for unexpected failures.

Specified by:

getFailureCallback in interface ResponseProcessor

Parameters:

druidQuery - The query associated with this failure

Returns:

The callback handler

getErrorCallback

public HttpErrorCallback getErrorCallback(DruidAggregationQuery<?> druidQuery)

Description copied from interface: ResponseProcessor

Callback for handling http errors.

Specified by:

getErrorCallback in interface ResponseProcessor

Parameters:

druidQuery - The query associated with this error

Returns:

The callback handler

processResponse

public void processResponse(com.fasterxml.jackson.databind.JsonNode json,
                            DruidAggregationQuery<?> query,
                            LoggingContext metadata)

If status code is 200, do the following

1. Extract uncoveredIntervalsOverflowed from X-Druid-Response-Context inside the JsonNode passed into DruidPartialDataResponseProcessor::processResponse, if it is true, invoke error response saying limit overflowed,
2. Extract uncoveredIntervals from X-Druid-Response-Contex inside the JsonNode passed into DruidPartialDataResponseProcessor::processResponse,
3. Parse both the uncoveredIntervals extracted above and allAvailableIntervals extracted from the union of all the query's datasource's availabilities from DataSourceMetadataService into SimplifiedIntervalLists,
4. Compare both SimplifiedIntervalLists above, if allAvailableIntervals has any overlap with uncoveredIntervals, invoke error response indicating druid is missing some data that are we are expecting to exists.

Specified by:

processResponse in interface ResponseProcessor

Parameters:

json - The json representing a druid data response

query - The query with the schema for processing this response

metadata - The LoggingContext to use