Data Slice Generator¶
The data slice generator is the underlying function used to generate data slices for the labeling function. If the label maker raises an error during the search or the output labels don’t seem correct, you need to check the logic in the labeling function or inspect the data for inherent errors. The data slice generator can help identify those problems. You also want to use the generator during the development of your labeling function as a best practice. But it is an optional step and not required to generate labels.
In this guide, use the data slice generator to inspect data slices and apply a labeling function. To get started, load a mock dataset of transactions and sample the data to see how the transactions look.
[1]:
import composeml as cp
[2]:
df = cp.demos.load_transactions()
df = df[df.columns[:7]]
df.sample(n=5, random_state=0)
[2]:
| transaction_id | session_id | transaction_time | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|---|
| 26 | 94 | 24 | 2014-01-01 05:55:20 | 5 | 100.42 | 5 | tablet |
| 86 | 274 | 7 | 2014-01-01 01:46:10 | 5 | 14.45 | 3 | tablet |
| 2 | 495 | 1 | 2014-01-01 00:14:05 | 5 | 69.45 | 2 | desktop |
| 55 | 275 | 4 | 2014-01-01 00:45:30 | 5 | 108.11 | 1 | mobile |
| 75 | 368 | 27 | 2014-01-01 06:36:30 | 5 | 139.43 | 1 | mobile |
Labeling Function¶
Define a labeling function that returns how much a customer spent given a slice of transactions.
[3]:
def total_spent(df):
return df['amount'].sum()
Data Slices¶
The LabelMaker.slice() method creates the data slice generator. The parameters of this method can be passed directly to LabelMaker.search() to generate the labels. In the following sections, we explain how to use the data slice generator to make data slices consecutive, overlap, or spread out.
See also
For a conceptual explanation of the process, see Main Concepts.
Consecutive¶
When the gap size is equal to the window size, the data slices are consecutive. In other words, the data slices do not overlap and are not spread out (in other words, don’t skip any data). This is the default value for the gap size. To demonstrate this example, generate data slices using these parameters.
To start, create a label maker with the 2-hour window size.
[4]:
lm = cp.LabelMaker(
target_dataframe_index="customer_id",
time_index="transaction_time",
labeling_function=total_spent,
window_size="2h",
)
Next, create a data slice generator with the 2-hour gap size. The default value for the gap size is the window size.
Tip
You can directly set minimum_data as the first cutoff time.
[5]:
slices = lm.slice(
df.sort_values('transaction_time'),
num_examples_per_instance=-1,
minimum_data='2014-01-01',
)
Consecutive - Data Slice #1¶
By printing this data slice, you can see that that it is the first slice of transactions (denoted by the slice_number) for customer 1. This data slice contains all of the customer’s transactions that occurred within the 2-hour window between 2014-01-01 00:00:00 and 2014-01-01 02:00:00. You can also see that the 2-hour gap aligns the cutoff times to the window, so the next data slice starts at the end of this data slice.
[6]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 1
slice_start 2014-01-01 00:00:00
slice_stop 2014-01-01 02:00:00
next_start 2014-01-01 02:00:00
[6]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 00:45:30 | 275 | 4 | 5 | 108.11 | 1 | mobile |
| 2014-01-01 00:46:35 | 101 | 4 | 5 | 112.53 | 1 | mobile |
| 2014-01-01 00:47:40 | 80 | 4 | 5 | 6.29 | 1 | mobile |
| 2014-01-01 00:52:00 | 163 | 4 | 5 | 31.37 | 1 | mobile |
| 2014-01-01 00:53:05 | 293 | 4 | 5 | 82.88 | 1 | mobile |
| 2014-01-01 00:57:25 | 103 | 4 | 5 | 20.79 | 1 | mobile |
| 2014-01-01 01:03:55 | 488 | 4 | 5 | 129.00 | 1 | mobile |
| 2014-01-01 01:05:00 | 413 | 4 | 5 | 119.98 | 1 | mobile |
| 2014-01-01 01:31:00 | 191 | 6 | 5 | 139.23 | 1 | tablet |
| 2014-01-01 01:37:30 | 372 | 6 | 5 | 114.84 | 1 | tablet |
| 2014-01-01 01:38:35 | 387 | 6 | 5 | 49.71 | 1 | tablet |
Apply the labeling function for the total amount spent on this data slice.
[7]:
total_spent(ds)
[7]:
914.7300000000001
Consecutive - Data Slice #2¶
In the second data slice, you can see the next 2 consecutive hours of transactions between 2014-01-01 02:00:00 and 2014-01-01 04:00:00. This is useful for generating labels that consecutively process the data only once.
[8]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 2
slice_start 2014-01-01 02:00:00
slice_stop 2014-01-01 04:00:00
next_start 2014-01-01 04:00:00
[8]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 02:28:25 | 287 | 9 | 5 | 50.94 | 1 | desktop |
| 2014-01-01 03:29:05 | 190 | 14 | 5 | 110.52 | 1 | tablet |
| 2014-01-01 03:39:55 | 7 | 14 | 5 | 107.42 | 1 | tablet |
Apply our labeling function for the total amount spent on this data slice.
[9]:
total_spent(ds)
[9]:
268.88
Overlap¶
When the the gap size is less than the window size, the data slices overlap. You can use this for rolling window based labeling processes. The amount of overlap is the difference between the window and gap size. For example, if the window size is 3 hours and the gap size is 1 hour, then 2 hours will overlap on each data slice. To demonstrate this example, generate data slices using these parameters.
To start, create a label maker with the 3-hour window size.
[10]:
lm = cp.LabelMaker(
target_dataframe_index="customer_id",
time_index="transaction_time",
labeling_function=total_spent,
window_size="3h",
)
Next, create a data slice generator with the 1-hour gap size.
[11]:
slices = lm.slice(
df.sort_values('transaction_time'),
num_examples_per_instance=-1,
minimum_data='2014-01-01',
gap="1h",
)
Overlap - Data Slice #1¶
The first data slice contains all of the customer’s transactions that occurred within the 3-hour window between 2014-01-01 00:00:00 and 2014-01-01 03:00:00. The 1-hour gap spaces apart the cutoff time of this data slice at 2014-01-01 00:00:00 from the cutoff time of the next data slice at 2014-01-01 01:00:00.
[12]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 1
slice_start 2014-01-01 00:00:00
slice_stop 2014-01-01 03:00:00
next_start 2014-01-01 01:00:00
[12]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 00:45:30 | 275 | 4 | 5 | 108.11 | 1 | mobile |
| 2014-01-01 00:46:35 | 101 | 4 | 5 | 112.53 | 1 | mobile |
| 2014-01-01 00:47:40 | 80 | 4 | 5 | 6.29 | 1 | mobile |
| 2014-01-01 00:52:00 | 163 | 4 | 5 | 31.37 | 1 | mobile |
| 2014-01-01 00:53:05 | 293 | 4 | 5 | 82.88 | 1 | mobile |
| 2014-01-01 00:57:25 | 103 | 4 | 5 | 20.79 | 1 | mobile |
| 2014-01-01 01:03:55 | 488 | 4 | 5 | 129.00 | 1 | mobile |
| 2014-01-01 01:05:00 | 413 | 4 | 5 | 119.98 | 1 | mobile |
| 2014-01-01 01:31:00 | 191 | 6 | 5 | 139.23 | 1 | tablet |
| 2014-01-01 01:37:30 | 372 | 6 | 5 | 114.84 | 1 | tablet |
| 2014-01-01 01:38:35 | 387 | 6 | 5 | 49.71 | 1 | tablet |
| 2014-01-01 02:28:25 | 287 | 9 | 5 | 50.94 | 1 | desktop |
Apply our labeling function for the total spent on this data slice.
[13]:
total_spent(ds)
[13]:
965.6700000000001
Overlap - Data Slice #2¶
In the second data slice, there is a 2-hour overlap on the transactions that occurred between 2014-01-01 01:00:00 and 2014-01-01 03:00:00. By adjusting the gap size, you can set the precise amount of overlap in the data slices. That is useful for generating labels with specific overlap.
[14]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 2
slice_start 2014-01-01 01:00:00
slice_stop 2014-01-01 04:00:00
next_start 2014-01-01 02:00:00
[14]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 01:03:55 | 488 | 4 | 5 | 129.00 | 1 | mobile |
| 2014-01-01 01:05:00 | 413 | 4 | 5 | 119.98 | 1 | mobile |
| 2014-01-01 01:31:00 | 191 | 6 | 5 | 139.23 | 1 | tablet |
| 2014-01-01 01:37:30 | 372 | 6 | 5 | 114.84 | 1 | tablet |
| 2014-01-01 01:38:35 | 387 | 6 | 5 | 49.71 | 1 | tablet |
| 2014-01-01 02:28:25 | 287 | 9 | 5 | 50.94 | 1 | desktop |
| 2014-01-01 03:29:05 | 190 | 14 | 5 | 110.52 | 1 | tablet |
| 2014-01-01 03:39:55 | 7 | 14 | 5 | 107.42 | 1 | tablet |
Apply our labeling function for the total spent on this data slice.
[15]:
total_spent(ds)
[15]:
821.6400000000001
Spread Out¶
When the the gap size is greater than the window size, there is data in-between data slices that is skipped. You can use this for labeling data at specific intervals of time. The amount of data skipped is the difference between the gap and window size. For example, if the gap size is 3 hours and the window size is 1 hour, then 2 hours of data will be skipped in-between data slices. To demonstrate this example, generate data slices using these parameters.
To start, create a label maker with the 1-hour window size.
[16]:
lm = cp.LabelMaker(
target_dataframe_index="customer_id",
time_index="transaction_time",
labeling_function=total_spent,
window_size="1h",
)
Next, create a data slice generator with the 3-hour gap size.
[17]:
slices = lm.slice(
df.sort_values('transaction_time'),
num_examples_per_instance=-1,
minimum_data='2014-01-01',
gap="3h",
)
Spread Out - Data Slice #1¶
The first data slice contains all of the customer’s transactions that occurred within the 1-hour window between 2014-01-01 00:00:00 and 2014-01-01 01:00:00. The 3-hour gap spaces apart the cutoff time of this data slice at 2014-01-01 00:00:00 from the cutoff time of the next data slice at 2014-01-01 03:00:00.
[18]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 1
slice_start 2014-01-01 00:00:00
slice_stop 2014-01-01 01:00:00
next_start 2014-01-01 03:00:00
[18]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 00:45:30 | 275 | 4 | 5 | 108.11 | 1 | mobile |
| 2014-01-01 00:46:35 | 101 | 4 | 5 | 112.53 | 1 | mobile |
| 2014-01-01 00:47:40 | 80 | 4 | 5 | 6.29 | 1 | mobile |
| 2014-01-01 00:52:00 | 163 | 4 | 5 | 31.37 | 1 | mobile |
| 2014-01-01 00:53:05 | 293 | 4 | 5 | 82.88 | 1 | mobile |
| 2014-01-01 00:57:25 | 103 | 4 | 5 | 20.79 | 1 | mobile |
Apply our labeling function for the total spent on this data slice.
[19]:
total_spent(ds)
[19]:
361.96999999999997
Spread Out - Data Slice #2¶
In the second data slice, you can see that 2 hours of transactions were skipped between 2014-01-01 01:00:00 and 2014-01-01 03:00:00. By adjusting the gap size, you can set the precise amount of data to skip in-between data slices. That is useful for generating labels that target specific portions of a dataset.
[20]:
ds = next(slices)
print(ds.context)
ds
customer_id 1
slice_number 2
slice_start 2014-01-01 03:00:00
slice_stop 2014-01-01 04:00:00
next_start 2014-01-01 06:00:00
[20]:
| transaction_id | session_id | product_id | amount | customer_id | device | |
|---|---|---|---|---|---|---|
| transaction_time | ||||||
| 2014-01-01 03:29:05 | 190 | 14 | 5 | 110.52 | 1 | tablet |
| 2014-01-01 03:39:55 | 7 | 14 | 5 | 107.42 | 1 | tablet |
Apply the labeling function for the total spent on this data slice.
[21]:
total_spent(ds)
[21]:
217.94
Data Slice Context¶
Each data slice has a context attribute to access its metadata. That is useful for integrating the context with the logic in the labeling function.
[22]:
vars(ds.context)
[22]:
{'next_start': Timestamp('2014-01-01 06:00:00'),
'slice_stop': Timestamp('2014-01-01 04:00:00'),
'slice_start': Timestamp('2014-01-01 03:00:00'),
'slice_number': 2,
'customer_id': 1}